Skip to content

fix(codex): kernel-assigned fixture ports close the EADDRINUSE port-theft flake - #811

Merged
danshapiro merged 1 commit into
mainfrom
fix/codex-fixture-port-file
Sep 20, 2026
Merged

danshapiro merged 1 commit into
mainfrom
fix/codex-fixture-port-file

Conversation

@danshapiro

Copy link
Copy Markdown
Owner

The flake

sidecar_reconcile::tests::reattach_shutdown_kills_only_after_reverification panicked once with live child has a starttime under a full workspace run (96-core host), passed 180/180 on rerun. Still live on main — #805's sixpack touched freshagent fixtures, not this suite.

Root cause (verified empirically, not just theorized)

The fixture-spawn helper pre-allocated its listening port via bind-then-drop, then spawned node (~100ms boot window). This kernel's bind(0) allocator is randomized and measurably re-issues just-freed ports (142 duplicates in 2000 allocations). A sibling test's fixture could be handed the freed port first: our child died of EADDRINUSE (tokio reaps it, /proc gone) while our spawn probe happily completed a WS handshake with the THIEF — the same fixture protocol, indistinguishable — until record_for_child read /proc evidence of our dead child and panicked. fd exhaustion ruled out (host limit 1M); zombie path ruled out (tokio auto-reaps).

Fix (structural, not a retry)

The fake app-server fixture now binds port 0 — kernel-atomic, nothing can steal it — and reports the actual port through a port-file (additive env-gated mode, FAKE_CODEX_APP_SERVER_PORT_FILE; the durable-writer submode already used this exact pattern). The helper waits for the report, then probes. Evidence-capture panics now name the pid and likely cause, so any future failure is one-look diagnosable. A 12-way concurrent-spawn regression test pins every child live + every port distinct.

Verification

  • freshell-codex: 181/181 (lib, featured) — ×10 stress runs; plus all 8 integration binaries green in the gate's workspace run (266 tests total)
  • Hermetic routing tests 26/26 (both gate-fix PRs)
  • fmt / clippy / typecheck green in the gate run
  • Pre-push gate note: the workspace lane was blocked once by a single freshagent flake — session_handoff::tests::the_claude_handoff_target_binding_carries_the_handoff_generation (None vs Some(2), order/state-sensitive; same family test(fresh-agent): deflake the b8ke handoff backoff window test #808 defalked a sibling of). It passes 3/3 solo, in-module (76/76), and on main; my diff touches no freshagent code. Pushed with the documented --no-verify for that flake only; CI rust-gate re-runs the identical workspace suite here as the authoritative backstop. The handoff flake deserves the same fake-timers deflaking treatment test(fresh-agent): deflake the b8ke handoff backoff window test #808 gave its sibling — flagged for follow-up.

…heft flake

The fixture-spawn helper pre-allocated its listening port via bind-drop,
leaving a window where the kernel hands the just-freed port to a sibling
test's fixture: ours dies EADDRINUSE (tokio reaps it, /proc gone) while
the spawn probe happily handshakes with the thief — the same fixture
protocol, indistinguishable — until /proc evidence capture panics with
"live child has a starttime" (observed once, under a full workspace
run on a 96-core host; fd-exhaustion ruled out, ulimit -n = 1M, and the
kernel's bind(0) allocator re-issues just-freed ports measurably).

The fixture now binds port 0 (kernel-atomic, no free-port window) and
reports the real port through a port file; the helper waits for the
report, then probes. Evidence-capture panics now name the pid and the
likely cause. A 12-way concurrent-spawn regression test pins every child
live + every port distinct.
@danshapiro
danshapiro merged commit 55a036d into main Sep 20, 2026
7 checks passed
@danshapiro
danshapiro deleted the fix/codex-fixture-port-file branch September 20, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant