Skip to content

fix(web-search): preserve quota evidence when Retry-After would outlive sidecar deadline - #568

Closed
luvs01 wants to merge 4 commits into
devfrom
codex/fix-429-retry-handling-to-preserve-quota
Closed

luvs01 wants to merge 4 commits into
devfrom
codex/fix-429-retry-handling-to-preserve-quota

Conversation

@luvs01

@luvs01 luvs01 commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Prevent laundering an observed HTTP 429 (quota refusal) into a timeout by ensuring a computed Retry-After delay actually fits the remaining sidecar deadline before sleeping to replay. This preserves quota evidence used by account-pool routing.

Description

  • Update the sidecar 429 retry loop in runWebSearch to reject a replay when delay exceeds RETRY_AFTER_CEILING_MS or is greater-or-equal to the remaining sidecar lifetime computed from settings.timeoutMs - (Date.now() - t0), so the original 429 is returned and recorded.【F:src/web-search/executor.ts†L123-L136】【F:src/web-search/executor.ts†L47-L55】
  • Add a focused regression test that asserts a Retry-After that cannot fit the remaining sidecar deadline preserves the 429 and records it as such.【F:tests/web-search/web-search-sidecar-429.test.ts†L82-L92】
  • Update runtime and provider-adapters documentation to record the deadline-aware 429-retry contract so architecture docs reflect the behavior.【F:structure/runtime.md†L356-L358】【F:structure/providers-and-adapters.md†L153-L155】

Testing

  • ✅ Ran the focused web-search tests with npm exec --yes --package bun@1.4.0 -- bun test tests/web-search/web-search-sidecar-429.test.ts, which passed (4 tests).
  • ✅ Ran static checks and repo validations with npm exec --yes --package bun@1.4.0 -- bun run typecheck, npm exec --yes --package bun@1.4.0 -- bun run structure:check, and npm exec --yes --package bun@1.4.0 -- bun run privacy:scan, which all succeeded.
  • ⚠️ Ran the full suite via npm exec --yes --package bun@1.4.0 -- bun run test; the run completed but exited non-zero with many unrelated environment- and timing-sensitive failures (process-control and network/proxy fixtures); the focused regression and static checks relevant to this change are green.

Codex Task


Devin Review

@coderabbitai

coderabbitai Bot commented Sep 20, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository: luvs01/opencodex/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: f616cba8-251d-4866-8e06-3e6bec2e8fe3


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 20, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-20T12:53:11.935309Z d85a5d8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown

Deterministic PR hygiene checks passed.

devin-ai-integration[bot]

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

@github-actions github-actions Bot added the bug Something isn't working label Sep 20, 2026
lidge-jun#5334 folds developer to system for destinations that have not recorded
whether they accept the role. Two suites still asserted the forwarded role
on an unrecorded fixture: a document-carrying developer turn and the
dangling tool_calls barrier placement. Declare foldDeveloperRoleToSystem:
false on their destinations, matching the ordering suites, and pin the
unrecorded fold on the document path.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown

CI fix pushed (d38aecb2):

Both files verified locally against the real adapter (all 9 dangling-toolcalls tests pass; the document-turn assertions produce developer/system exactly as expected). The same stale assertions fail on dev today, so merging this PR also fixes dev.

The pre-retry delay check budgets only the backoff, but the body release
can consume up to 1s first, so the sidecar deadline can fire during
cleanup or mid-sleep. That threw TimeoutError into the shared catch and
recorded timeout, laundering the observed 429 this path exists to
preserve. Catch the wait: a caller abort still propagates to the shared
catch (connect_neutral), while a deadline expiry breaks with the 429 in
hand. Also start t0 before the deadline timer so the remaining-time
check stays conservative.

Co-Authored-By: Epinephrine <luvs01@hanmail.net>
@devin-ai-integration

Copy link
Copy Markdown

Fixed in f024b7c. The release+backoff wait is now wrapped so a sidecar deadline firing during releaseResponseBodyBestEffort or sleepWithAbort breaks the loop with the observed 429, which flows to recordOutcome(429) instead of being laundered into timeout by the shared catch. Caller aborts still propagate (recorded connect_neutral). Also moved t0 ahead of signalWithTimeout so the remaining-time check is conservative rather than optimistic, and added a regression test with a never-settling cancel() covering the cleanup-overhead path.

@devin-ai-integration

Copy link
Copy Markdown

Fixed in f024b7c — same change as the thread above: a deadline expiry during body release or the backoff sleep now ends the replay loop with the existing 429 response (recorded as 429) rather than throwing into the shared catch as timeout; a caller abort still propagates. Regression test added with a never-settling cancel() body.

@luvs01

luvs01 commented Sep 21, 2026

Copy link
Copy Markdown
Owner Author

Closing: superseded by combined PR #614, which carries this PR's deadline-safe quota evidence preservation together with the replay-cache isolation on current dev. Its unrecorded-destination test assertion was adapted to the current oldDeveloperRoleToSystem fixture semantics. Combined tests: 65 pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aardvark bug Something isn't working codex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant