fix(channels): back off exponentially on feishu ws reconnect - #375
fix(channels): back off exponentially on feishu ws reconnect#375LivXue wants to merge 2 commits into
Conversation
The reconnect loop retried on a fixed 5s sleep forever. lark-oapi's ws Client.start() blocks while the connection lives and only returns by raising, almost always a ClientException (bad credentials or conn limit exceeded), so the old loop hammered Feishu's auth endpoint every 5s on permanent failures. Retries now use exponential backoff capped at 300s, and stop() can interrupt the sleep early. Co-authored-by: Claude (deepseek-v4-pro) <noreply@anthropic.com>
gloryfromca
left a comment
There was a problem hiding this comment.
Blocking: the reconnect loop can terminate itself after sustained failures; see the inline note.
I reviewed the full diff at this head and tried to refute the finding against the actual lark-oapi 1.5.3 control flow. Coverage included AGENTS.md, CLAUDE.md, and CONTEXT-MAP.md/CONTEXT.md; the changed implementation and tests; Feishu start/stop callers; relevant file history; backward compatibility; and whether tests were weakened (they were not; the patch only adds coverage).
Verification: uv run --extra channel-feishu pytest tests/test_channels_feishu.py -q passed with 38 tests and one pre-existing coroutine warning. uv run --extra channel-feishu ruff check raven/channels/adapters/feishu/channel.py tests/test_channels_feishu.py and ruff format --check both passed. I also reproduced the overflow by driving _run_ws_supervised() with immediate failures: it raised after 1,024 completed backoffs.
| time.sleep(5) | ||
| delay = min( | ||
| _RECONNECT_BACKOFF_MAX_S, | ||
| _RECONNECT_BACKOFF_INITIAL_S * (_RECONNECT_BACKOFF_FACTOR**attempt), |
There was a problem hiding this comment.
Blocking: the unbounded attempt exponent eventually overflows and terminates the reconnect supervisor. After 1,024 failed starts, 2.0**attempt raises OverflowError before min() can apply the 300-second cap; at the configured schedule this is roughly 85 hours of a sustained failure such as the SDK's connection-limit ClientException. The daemon thread then exits while _running remains true, so the channel silently stops reconnecting. Please cap attempt once the maximum delay is reached (or carry the already-capped delay forward) and add a regression case beyond the cap.
There was a problem hiding this comment.
Fixed. Dropped the attempt exponent entirely and carry the already-capped delay forward (delay = min(300, delay * 2)), so sustained failures can no longer overflow the backoff math; the supervisor thread keeps reconnecting indefinitely.
Added test_reconnect_backoff_never_overflows_beyond_cap, which drives 2,000 consecutive ClientException failures and asserts the ladder reaches the cap then stays flat without raising.
Verified locally: uv run pytest tests/test_channels_feishu.py -q -> 39 passed, ruff check and ruff format --check clean.
There was a problem hiding this comment.
Confirmed on head 01411cc1418e: carrying the capped delay preserves the 5/10/20/40/80/160/300-second ladder and remains at 300 seconds indefinitely. The new 2,000-failure regression passes, and an additional 100,000-step stress check stayed capped without raising. The Feishu suite reports 39 passed with the same pre-existing coroutine warning. This resolves my blocker.
The per-attempt exponent 2.0**attempt overflows after 1,024 failed starts, which raises OverflowError outside the try blocks and kills the reconnect thread while _running stays true. Grow the delay iteratively instead and clamp it to the cap, so sustained failures never overflow the backoff math. Co-authored-by: Claude (deepseek-v4-pro) <noreply@anthropic.com>
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
I reviewed the fix delta and rechecked the resulting full diff against the prior review. The iterative capped delay removes the overflow path while preserving retry timing and interruptible shutdown, and the new stress regression covers the sustained-failure case without weakening existing tests. I also rechecked the applicable repository rules, Feishu lifecycle callers and SDK behavior, relevant history, and backward compatibility; no new issue was found.
Verification: uv run --extra channel-feishu pytest tests/test_channels_feishu.py -q passed with 39 tests and one pre-existing coroutine warning. Ruff check and format-check passed for both changed files. A separate 100,000-step delay calculation remained capped at 300 seconds.
Summary
The Feishu adapter's reconnect loop retried on a fixed 5s sleep with no
backoff and no attempt cap. lark-oapi's ws Client.start() blocks while
the connection lives (the SDK reconnects transient drops internally with
its own 120s interval) and only returns by raising, almost always a
ClientException such as bad credentials or connection-limit exceeded.
The old loop therefore hammered Feishu's auth endpoint every 5s on
permanent failures.
Changes:
2): 5s, 10s, 20s, 40s, 80s, 160s, 300s.
warning.
backoff promptly (no lingering thread).
Three tests added covering ladder growth/cap, ClientException handling,
and stop() interrupting the sleep.
Type
Verification
Commands and results:
uv run pytest tests/test_channels_feishu.py -q-> 38 passed(1 warning is pre-existing, from test_stop_blocks_zombie_inbound).
uv run ruff check raven/channels/adapters/feishu/channel.py tests/test_channels_feishu.py-> All checks passed.uv run ruff format raven/channels/adapters/feishu/channel.py tests/test_channels_feishu.py-> 2 files reformatted (applied).check.
No user-facing docs affected: this is an internal retry-cadence change
with no config or interface change.
Risk
No security surface change: only retry timing is affected. No config,
CLI, or API change; existing deployments behave the same apart from
slower retries after repeated connection failures. Rollback: revert the
single commit.
Related Issues
N/A