Skip to content

fix(channels): back off exponentially on feishu ws reconnect - #375

Open
LivXue wants to merge 2 commits into
mainfrom
fix/feishu_reconnect_backoff
Open

fix(channels): back off exponentially on feishu ws reconnect#375
LivXue wants to merge 2 commits into
mainfrom
fix/feishu_reconnect_backoff

Conversation

@LivXue

@LivXue LivXue commented Aug 29, 2026

Copy link
Copy Markdown
Member

Summary

The Feishu adapter's reconnect loop retried on a fixed 5s sleep with no
backoff and no attempt cap. lark-oapi's ws Client.start() blocks while
the connection lives (the SDK reconnects transient drops internally with
its own 120s interval) and only returns by raising, almost always a
ClientException such as bad credentials or connection-limit exceeded.
The old loop therefore hammered Feishu's auth endpoint every 5s on
permanent failures.

Changes:

  • Retries now use exponential backoff capped at 300s (5s initial, factor
    2): 5s, 10s, 20s, 40s, 80s, 160s, 300s.
  • ClientException is logged as a permanent error instead of a generic
    warning.
  • The backoff sleep runs in 1s slices so stop() can interrupt a long
    backoff promptly (no lingering thread).

Three tests added covering ladder growth/cap, ClientException handling,
and stop() interrupting the sleep.

Type

  • Fix
  • Feature
  • Docs
  • CI / tooling
  • Refactor
  • Other

Verification

  • Relevant tests pass locally
  • Relevant lint / type checks pass locally
  • User-facing docs or screenshots are updated when needed

Commands and results:

  • uv run pytest tests/test_channels_feishu.py -q -> 38 passed
    (1 warning is pre-existing, from test_stop_blocks_zombie_inbound).
  • uv run ruff check raven/channels/adapters/feishu/channel.py tests/test_channels_feishu.py -> All checks passed.
  • uv run ruff format raven/channels/adapters/feishu/channel.py tests/test_channels_feishu.py -> 2 files reformatted (applied).
  • pre-commit hooks all passed, including the conventional commit message
    check.

No user-facing docs affected: this is an internal retry-cadence change
with no config or interface change.

Risk

  • Security impact considered
  • Backward compatibility considered
  • Rollback path is clear for risky changes

No security surface change: only retry timing is affected. No config,
CLI, or API change; existing deployments behave the same apart from
slower retries after repeated connection failures. Rollback: revert the
single commit.

Related Issues

N/A

The reconnect loop retried on a fixed 5s sleep forever. lark-oapi's
ws Client.start() blocks while the connection lives and only returns
by raising, almost always a ClientException (bad credentials or conn
limit exceeded), so the old loop hammered Feishu's auth endpoint every
5s on permanent failures. Retries now use exponential backoff capped
at 300s, and stop() can interrupt the sleep early.

Co-authored-by: Claude (deepseek-v4-pro) <noreply@anthropic.com>

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the reconnect loop can terminate itself after sustained failures; see the inline note.

I reviewed the full diff at this head and tried to refute the finding against the actual lark-oapi 1.5.3 control flow. Coverage included AGENTS.md, CLAUDE.md, and CONTEXT-MAP.md/CONTEXT.md; the changed implementation and tests; Feishu start/stop callers; relevant file history; backward compatibility; and whether tests were weakened (they were not; the patch only adds coverage).

Verification: uv run --extra channel-feishu pytest tests/test_channels_feishu.py -q passed with 38 tests and one pre-existing coroutine warning. uv run --extra channel-feishu ruff check raven/channels/adapters/feishu/channel.py tests/test_channels_feishu.py and ruff format --check both passed. I also reproduced the overflow by driving _run_ws_supervised() with immediate failures: it raised after 1,024 completed backoffs.

time.sleep(5)
delay = min(
_RECONNECT_BACKOFF_MAX_S,
_RECONNECT_BACKOFF_INITIAL_S * (_RECONNECT_BACKOFF_FACTOR**attempt),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the unbounded attempt exponent eventually overflows and terminates the reconnect supervisor. After 1,024 failed starts, 2.0**attempt raises OverflowError before min() can apply the 300-second cap; at the configured schedule this is roughly 85 hours of a sustained failure such as the SDK's connection-limit ClientException. The daemon thread then exits while _running remains true, so the channel silently stops reconnecting. Please cap attempt once the maximum delay is reached (or carry the already-capped delay forward) and add a regression case beyond the cap.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Dropped the attempt exponent entirely and carry the already-capped delay forward (delay = min(300, delay * 2)), so sustained failures can no longer overflow the backoff math; the supervisor thread keeps reconnecting indefinitely.

Added test_reconnect_backoff_never_overflows_beyond_cap, which drives 2,000 consecutive ClientException failures and asserts the ladder reaches the cap then stays flat without raising.

Verified locally: uv run pytest tests/test_channels_feishu.py -q -> 39 passed, ruff check and ruff format --check clean.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed on head 01411cc1418e: carrying the capped delay preserves the 5/10/20/40/80/160/300-second ladder and remains at 300 seconds indefinitely. The new 2,000-failure regression passes, and an additional 100,000-step stress check stayed capped without raising. The Feishu suite reports 39 passed with the same pre-existing coroutine warning. This resolves my blocker.

The per-attempt exponent 2.0**attempt overflows after 1,024 failed
starts, which raises OverflowError outside the try blocks and kills
the reconnect thread while _running stays true. Grow the delay
iteratively instead and clamp it to the cap, so sustained failures
never overflow the backoff math.

Co-authored-by: Claude (deepseek-v4-pro) <noreply@anthropic.com>

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

I reviewed the fix delta and rechecked the resulting full diff against the prior review. The iterative capped delay removes the overflow path while preserving retry timing and interruptible shutdown, and the new stress regression covers the sustained-failure case without weakening existing tests. I also rechecked the applicable repository rules, Feishu lifecycle callers and SDK behavior, relevant history, and backward compatibility; no new issue was found.

Verification: uv run --extra channel-feishu pytest tests/test_channels_feishu.py -q passed with 39 tests and one pre-existing coroutine warning. Ruff check and format-check passed for both changed files. A separate 100,000-step delay calculation remained capped at 300 seconds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants