readiness: Codex re-auth failure takes healthy multi-provider proxy out of service

[nordz0r]

Client or integration

Direct HTTP/API client

Area

Service lifecycle

Summary

When one or more Codex OAuth pool accounts require re-authentication, OpenCodex reports /readyz as HTTP 503 even though the proxy process is running and other configured providers/routes remain usable. In a single-replica Kubernetes deployment, the readiness failure removes the only Service endpoint and makes the entire proxy inaccessible.

Expected: Codex account-pool degradation should be reported as a component/provider failure. The proxy should remain ready when its HTTP server is healthy and at least one route/provider can serve traffic; native Codex traffic can fail or be marked degraded independently.

Reproduction

1. Run OpenCodex in a container/Kubernetes pod with multiple providers and a Codex account pool.
2. Leave one or more Codex accounts expired so they require re-authentication.
3. Start the proxy and query /healthz and /readyz.
4. Observe /healthz and the proxy listener become available, while /readyz returns 503 and Kubernetes marks the pod unready.
5. With a single replica, the Service has no ready endpoint even though non-Codex provider routes are configured.

Version

v2.59.0 (nordz0r/opencodex image)

Operating system

Debian 12 / Kubernetes (single-node k3s)

Provider and model

Codex account pool; other configured providers include Anthropic, Google Antigravity, OpenAI, and xAI routes.

Logs or error output

🚀 opencodex proxy running on http://localhost:10100
GET /healthz      → health check
GET /api/*        → management API
[opencodex] catalog sync: ... Codex accounts ... need reauthentication

GET /readyz -> 503
Kubernetes: Startup probe failed: HTTP probe failed with statuscode: 503
Pod: Running, 0/1 Ready, no process crash

After changing the Kubernetes probe to /healthz, the pod becomes 1/1 Ready; authenticated catalog and non-Codex inference checks succeed while the Codex pool remains partially degraded. This indicates the process is viable and the account-pool state is too broad a readiness gate.

Screenshots and supporting files

Not applicable.

Redacted configuration

{
  "deployment": "single replica",
  "readinessProbe": "/readyz",
  "livenessProbe": "/healthz",
  "codexPool": "multiple OAuth accounts, some requiring re-authentication",
  "otherRoutes": "configured non-Codex providers"
}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[lidge-jun]

Closed by #5199, squash-merged into dev as 09698b9b3a.

Your diagnosis was right and the mechanism turned out to be one layer over from where I first looked, so it is worth recording precisely.

/readyz was driven by the post-startup Codex catalog sync, and a nonempty warning in that sync's result was terminal for the gate alongside ok !== true. The two fields answer different questions. ok is the sync's verdict on whether the essential work succeeded — config injection and the write admission before it. warning names a degradation the sync itself decided to continue past: no catalog source so Codex keeps its native catalog, combos omitted, a history relabel left to Codex's own writer, or a caught refresh exception after which injection still runs and reports its own ok. None of those stops the process from accepting HTTP or routing to a provider, so none of them should have closed the gate. The fix is to read ok alone. A throw, a null, or ok !== true stay terminal, because none of those is a classified outcome — the startup path never reached a verdict, so the gate cannot claim one.

The correction: the "Codex accounts need reauthentication" line you saw is a bare console.warn and never reaches the structured warning field, and a pool account's refresh failure is caught per account and swallowed. A single expired pool account was therefore not what closed your gate. What does reach it is a native main credential refresh failure, which sits outside that per-account catch, rejects the whole entitlement gather, and becomes the catch-all "catalog sync skipped" warning that pinned the process un-ready forever. Same blast radius you reported, one layer up.

This boundary is not new, only extended. The Claude Code roster reconciliation in src/cli/claude-agent-startup-sync.ts already delays the ready transition without being permitted to fail it, on exactly this reasoning.

Two related changes landed in the same PR and are worth knowing about, because they make a degraded pool actionable instead of merely non-fatal. poolAccountDto no longer collapses three reauthentication causes into one boolean and guesses the reason from overlapping flags; the cause observed at the probe is carried through, so a WHAM 401 and a dead refresh grant are now distinguishable by the operator who has to act on one of them. That work is carried from #5191 with the author credited.

Your workaround of pointing the readiness probe at /healthz is no longer necessary. With this on dev, /readyz stays ready while the Codex pool is partially degraded, and native Codex traffic fails or degrades on its own.

Evidence is exact-head hosted CI at 3252e665bb2c40211702a5f794f38ace99f080df: 29 successful checks, zero failures — test 1/4 through 4/4, macos 1/2 (13751 tests) and macos 2/2 (13504 tests) all green, plus gates, structure gate, docker smoke, storage policy, api usage, and the keyring and npm-global matrices. No local suite was run in this lane.