[Enhancement]: Mid-stream OpenAI socket resets can break Codex account affinity

[luvs01]

Client or integration

Codex App

Area

Proxy and routing

Summary

A native OpenAI Responses stream can return HTTP 200 and partial SSE data, then fail while OpenCodex reads the upstream body with a socket reset such as ECONNRESET.

Codex correctly treats the resulting stream disconnected before completion: Transport error: network error: error decoding response body as retryable and replays the logical request. OpenCodex should not replay a partially delivered stream because that could duplicate tool calls or other side effects.

The routing side effect is incorrect: the passthrough inspection path converts a mid-stream body-read failure into synthetic status 502 and records it through recordCodexUpstreamOutcome. Repeated client retries or concurrent streams can then reach upstreamFailoverThreshold, soft-avoid a healthy credential, clear thread affinity, and promote another account even though no account-specific HTTP rejection occurred.

Expected behavior:

• Keep the synthetic mid-stream 502 in request diagnostics.
• Do not apply that synthetic transport terminal to account failure streak, cooldown, affinity, or active-account selection.
• Release only a quota-probe lease owned by the interrupted request.
• Continue recording actual HTTP 429/402/5xx and semantic SSE response.failed outcomes.
• Do not replay the partially delivered request inside OpenCodex; leave bounded retry to Codex.
• Preserve the selected account for the Codex retry unless an actual account-specific rejection occurs.
• Record enough correlation-safe metadata to distinguish upstream body-read failure, downstream pipe failure, client cancellation, and terminal delivery.

Relevant source:

• src/server/relay.ts: passthrough inspection maps body-read failure to a synthetic 502 terminal.
• src/server/responses/core.ts: native passthrough terminal recording forwards that synthetic status to account routing.
• src/codex/routing.ts: repeated transient outcomes can clear affinity and promote another account.
• structure/04_transports-and-sidecars.md: mid-stream SSE failures are intentionally not replayed by the proxy.

Required regression coverage:

• HTTP 200 SSE emits a partial delta, then its body throws ECONNRESET three times for the same thread.
• Each request remains observable as a synthetic mid-stream failure, but account health, affinity, and active selection remain unchanged.
• A held scoped or account-wide probe lease is released exactly once.
• An actual SSE response.failed with status 502 still records an account failure.
• A client cancellation remains distinct and does not become an upstream failure.
• No extra upstream request is sent by OpenCodex after partial delivery.

Related: #914 handles DNS/reachability failures before response headers; this issue handles failures after HTTP 200 and partial body delivery. #913 covers compact failover only after an actual upstream 429/402. The same client-visible error is tracked upstream in openai/codex#29087.

Reproduction

1. Configure OpenAI Codex-login pool routing and bind a thread to account A.
2. Return HTTP 200 with text/event-stream from the OpenAI upstream.
3. Deliver at least one valid SSE delta, then make the upstream body reader throw an ECONNRESET-shaped error before response.completed.
4. Let Codex retry the request several times or reproduce the same failure concurrently.

Current: OpenCodex records synthetic 502 account failures. At the configured threshold it can clear A's affinity or promote another account without an actual HTTP rejection.

Expected: Codex can retry, but OpenCodex keeps account routing state unchanged and never replays the partial stream itself.

Version

2.10.0; the relevant passthrough terminal and routing behavior remained on dev at c72acb3e368d2613c41d6d2b15a771895843f03f when checked on 2026-08-03.

Operating system

Windows 11 Pro 10.0.26200. The account-outcome classification issue is platform-independent; the observed resets occurred in the Windows native passthrough path.

Provider and model

OpenAI (Codex login / forward account pool); native Codex Responses streaming.

Logs or error output

stream disconnected before completion: Transport error: network error: error decoding response body
TypeError: The socket connection was closed unexpectedly
code: ECONNRESET

Screenshots and supporting files

None. Exact request IDs, thread IDs, account identifiers, quota values, and timestamps are intentionally omitted.

Redacted configuration

{
  "providers": {
    "openai": {
      "adapter": "openai-responses",
      "authMode": "forward",
      "codexAccountMode": "pool"
    }
  },
  "upstreamFailoverThreshold": 3
}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[lidge-jun]

Triaged, and the answer is more interesting than a yes or no. Two corrections and a reframe.

OpenCodex does not replay a partially delivered stream. Retry helpers operate before a Response is returned (src/lib/upstream-retry.ts:203-238), and the failed-tail relay documents and implements "not a resend" (src/server/relay.ts:98-137). Any logical replay you observed is the client's own subsequent request. That half of the report is already satisfied.

The routing side effect is real. consumeForInspection() maps an upstream body-read exception to a synthetic failed, 502 labeled transportPhase="mid_stream", terminalSource="synthetic" (src/server/relay.ts:988), and codexForwardTerminalOutcomeRecorder() (src/server/responses/core.ts:427-469) records it as an account failure. Enough of them soft-avoid the account and clear affinity.

But it was designed that way, deliberately. devlog/_fin/260722_issue_bug_sweep/030_patch_s_sticky_502.md introduced this exact behavior, and its test matrix lists upstream read rejection (200 후 socket reset) with the expected outcome transient 실패 기록, affinity 해제. The comment is still in the source. So this is not a bug that slipped in — it is a policy someone chose, and reversing it is a policy decision.

I spent four audit rounds trying to make it a local fix and failed each time. terminalSource: "synthetic" records who manufactured the terminal event, not who caused the failure — and the eager path's catch (src/server/relay-eager.ts:256) additionally wraps inspection and rewriting, so the same label can mean an upstream body failure or a local exception in our own relay.

The real question is whether post-200 transport reliability should affect account routing at all. Arguments both ways: every pool account shares the host, so rotating cannot repair a host-side drop — but account health tracks transient reliability generally, which is why ordinary 5xx responses count.

Recorded with #914 in devlog/_plan/260803_transport_attribution/000_plan.md, since they are the same question before and after HTTP 200. Leaving open until that policy is decided rather than shipping a guard that suppresses real signal.

[luvs01]

A new live reproduction on installed opencodex 2.10.0 confirms the routing side effect described here.

Observed on 2026-08-03 UTC, with account identifiers and request contents omitted:

• One ongoing conversation produced a burst of terminal HTTP 502 records between 12:34:31 and 12:36:24. Multiple records had a non-null firstOutputMs, so at least part of the burst occurred after output had started.
• Codex's structured log then recorded stream disconnected before completion: Transport error: network error: error decoding response body and an ordinary sampling retry against http://127.0.0.1:10100/v1/responses.
• During the same window, the OpenCodex usage log changed from the base openai provider label to a different privacy-safe pool label. That pool account returned 429 twice, followed by additional router-level 429 responses. Codex turns then terminated with exceeded retry limit, last status: 429 Too Many Requests.
• Codex continued to address only the localhost proxy and had no account-selection event. The provider/account transition exists only in OpenCodex records, so the rotation was proxy-side.
• Runtime configuration was fill-first with autoSwitchThreshold: 100. Ongoing affinity should not rotate for ordinary fill-first selection, which leaves the transient-failure path as the matching trigger. The installed routing code defaults upstreamFailoverThreshold to 3 and records synthetic mid-stream 502 as an account transient failure that can clear affinity and promote an alternate.
• After the incident, quota refresh showed the main account below its weekly limit while both added pool accounts were at 100%. This does not prove the cache knew those values at selection time, but it demonstrates the user-visible failure mode: a network/body-decode incident displaced an ongoing thread onto an account that returned 429, and client retries ended in a retry-limit failure.

This evidence does not establish that every post-200 transport failure should be account-neutral. It does establish that the current synthetic-502 policy can break ongoing affinity and turn a recoverable transport failure into account rotation plus 429 exhaustion. A regression test should cover both relay modes and assert that client cancellation remains neutral, while policy work separates host/transport health from credential/account health.

[lidge-jun]

Re-labeled bug → enhancement, keeping proxy, streaming, tools. The behavior you describe is real and reproduces; the reclassification is about what kind of change would resolve it, not about whether it happens.

The mid-stream reset clearing account affinity was introduced deliberately. devlog/_fin/260722_issue_bug_sweep/030_patch_s_sticky_502.md records the intended outcome as transient 실패 기록, affinity 해제 — a mid-stream socket reset is recorded as a transient failure precisely so account health stops treating that account as sticky. Escalated transient health then requires two successes to recover.

So the current behavior is the policy working as written, and changing it is a policy decision about how aggressively affinity should survive a transport failure. That is an enhancement request against a recorded design, not a defect against an intended one.

Two things worth knowing for whoever picks this up:

The synthetic/mid-stream label is not a shortcut to the answer. It records who manufactured the terminal event, not who caused the failure, and the eager path's catch wraps our own inspection and rewriting too — so keying the new behavior off it would produce a guard that looks right and misattributes.

#922 is adjacent but not this. It makes shared network failures account-neutral, which is a different question from whether a confirmed mid-stream reset should drop affinity.

Keeping it open with the label corrected so it sorts with the other accepted behavior changes rather than sitting in the bug queue.

[lidge-jun]

Closing after a policy review. What the report shows is real and was reproduced, but the behavior is the intended account-health policy: post-200 transport failures count against the account so that persistent upstream trouble rotates traffic away. Treating a mid-stream socket reset as never-account-attributable would mask genuinely unhealthy accounts, and terminalSource="synthetic" alone does not establish that the reset was network-local — especially on the eager relay path.

The right evolution here is a transport-attribution policy (classifying post-200 failures before they touch affinity), which is an enhancement, not a defect fix. Happy to reopen against a concrete attribution proposal, or a repro isolating a non-network cause for the resets.

[luvs01]

Concrete attribution proposal against dev 7797586a8899c673eab48886a490e85b480c6d72: keep post-200 transport reliability in account health, but distinguish a locally enforced relay budget from a transport failure before recording the account outcome.

I isolated one non-network case using the unchanged eager relay, its real SSE inspector, an identity rewrite, and createTranslatorBudget. All input is generated in memory; there is no fetch or socket. The first valid text delta is delivered in every case. The healthy and local-budget cases receive identical valid SSE chunks; only the local budget changes.

Case	Local budget	Reader errors	Relay result	onSynthetic arguments
Healthy control	8192 bytes	0	real completed; no synthetic failure	none
Local rewrite-buffer limit	128 bytes	0	translation_buffer_limit; budget overflow count 1	failed, no reason
Reader-rejection control	8192 bytes	1, injected ECONNRESET	upstream_reset; budget overflow count 0	failed, no reason

The first delta is 61 bytes and the next valid delta is 312 bytes. Every case finishes once and returns retained budget usage to zero. This is an isolated mechanism reproduction, not a claim about the original incident's cause or a real network capture. It directly observes the relay callbacks; the downstream account effect below is established from the current recorder code, rather than from a live account mutation.

The distinction is lost at the accounting boundary. The owned rewrite-budget reservation can raise the local error, and the failed tail already distinguishes its wire code. But the eager callback carries only failure kind/reason. The caller maps an unqualified synthetic failure to 502, which becomes a transient account outcome. Before streaming, the request-decode handler already maps this error type to 413.

Proposed narrow policy:

1. Carry first-failure stage (read, inspection, framing, rewrite) separately from attribution evidence. Neither stage nor terminalSource=synthetic proves a local cause.
2. The first explicit local category is a failure raised by the request-owned translator-budget reservation itself. Record that as a caller/local-limit outcome, proposed semantic status 413, while preserving the existing failed response and translation_buffer_limit wire code. This is an explicit policy choice: rotating accounts should not be the recovery action for the proxy's own per-request limit. A generic error string/code, malformed upstream payload, arbitrary inspection/rewrite exception, or upstream body rejection does not qualify.
3. Keep every unclassified failure and post-200 reader failure on the existing transient policy. Do not reuse connect_neutral or write a host failure: this is not pre-connection reachability evidence. Do not record 200, which would incorrectly clear prior health. The caller/neutral path only releases a matching owned probe lease and preserves failure streaks, cooldowns, affinity and active-account selection.
4. Preserve a real terminal already observed, first-cause precedence over cleanup failures, cancellation/drain bounds, generation/lease checks, and exactly one accounting outcome. No post-output replay is added.

The regression matrix would pair the real local-budget case with reader rejection and unclassified inspection/rewrite controls, then cover a later cleanup error, real terminal before delivery failure, client cancellation, and stale/foreign probe ownership. The small budget above is a test control; production limits would not change.

Would this specific local-limit exception and provenance boundary be an acceptable basis for reopening the enhancement? I have not changed routing policy or reopened the issue. Broader post-200 account-neutral behavior is not part of this proposal.