[Bug]: spawned Bun child processes stop producing output and never exit, on both macOS and Windows CI legs

[lidge-jun]

Client or integration

CI only (GitHub Actions hosted macOS runners); not a client-facing path.

Area

CI and release

Summary

Sharded macOS test legs stop producing output mid-suite and are killed at their job budget with zero reported failures. Across four occurrences the signature is identical: a test logs PASS, then the job emits nothing at all for 16 to 18 minutes until the runner cancels it, and Bun and tee are orphan-killed. This has been present since at least the 2.58.0 release round and was recorded there as capacity cancellation, which the evidence below does not support. A related but distinct failure mode appears where a hook timeout exists to stop it: the same shard reports a beforeEach/afterEach hook timed out for this test at just over 30s instead of going silent.

Two consequences. Releases have been cut without complete macOS evidence while believing macOS reds were infrastructure. And because the silent form produces a cancellation rather than a failure, it is indistinguishable at a glance from the genuine capacity cancellations that also occur, so it has survived several rounds of triage.

macos control hitting its 30-minute budget is a separate problem with a different cause and is tracked in #4905. Do not merge the two.

Reproduction

Occurs on hosted macOS runners during the sharded platform-macos legs. It is not reproducible locally and no local verification was attempted for this report.

Silent form, four occurrences:

1. Run 35277719805, job 105392401547, macos 1/2, at dev a0f611d. Last output 21:52:22, a PASS in tests/codex-integration/codex-inject-write-lock.test.ts case a held lock makes real injection report busy and write nothing. Silence for 16m15s until cancellation at 22:08:37.
2. PR fix(reauth): preserve polling through cancellation HTTP races #4945, macos 2/2. Last PASS full ordering ignores empty entries and accepts raw upstream ids with slashes at 21:53:21.617. Silence for 17m33s until cancellation at 22:10:54.
3. Run 35247708168 attempt 2, job 105303640713, macos 2, during the 2.58.0 round. 2514 pass / 0 fail, then 17m32.9s of silence after a catalog picker test until the 20-minute cancellation.
4. Run 35254182109 attempt 1, job 105313855869, macos 2. 2512 pass / 0 fail, then 17m19.1s of silence after full ordering ignores empty entries and accepts raw upstream ids with slashes — the same locus as occurrence 2.

Hook-timeout form, two occurrences on unrelated branches:

5. Run 35279139497, job 105396939652, macos 2/2 at cca06b1 (PR test(cli): measure cold status setup before timed projections #4948, which changes only tests/cli/cli-status-json.test.ts). server combo failover 030 activation matrix > connect cancellation wins with 499, no backup, warning, or cooldown at 30066.66ms, hook timeout. 12807 pass / 12 skip / 1 fail.
6. Job 105398860800, macos 2/2 at 50ecf06 (PR fix(responses): enforce configured total sends during recovery #4947). Identical case, 30050.44ms, hook timeout. 12810 pass / 12 skip / 1 fail.

Neither branch touches tests/server/server-combo-failover-e2e.test.ts.

Version

2.59.0 (dev), and retrospectively 2.58.0.

Operating system

macOS, GitHub-hosted runners (Apple Silicon; /opt/homebrew in the job environment).

Provider and model

Not provider-specific.

Logs or error output

(fail) server combo failover 030 activation matrix > connect cancellation wins with 499, no backup, warning, or cooldown [30066.66ms]
  ^ a beforeEach/afterEach hook timed out for this test.
 12807 pass
 12 skip
 1 fail
Ran 12820 tests across 668 files. [649.49s]
##[error]macOS suite failed (exit 1).

Screenshots and supporting files

Not applicable.

Redacted configuration

Not applicable; this is a CI-environment defect with no user configuration involved.

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• [Bug] CI: Linux test shards intermittently hang ~15 minutes and are killed, leaving an orphan bun process #1302

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[lidge-jun]

This is not macOS-specific, and the title now understates it. The same signature just appeared on Windows.

Run 35286915280, job 105421227726, windows 5/9, on branch codex/4943-history-preflight-wal-immutable-fallback (PR #4957). The case reset-credit auto-redeemer runtime (#822) > two processes reserve one durable id before either consume settles failed at 25294.71ms with timed out waiting for both consumes. The diagnostics are the part that matters:

a pid=4496 exit=null
stdout: 
stderr: 
b pid=5128 exit=null
stdout: 
stderr: 

Both child processes were spawned and assigned PIDs. Neither exited. Neither wrote a single byte to stdout or stderr across the full 25-second window. Every other case in that file passed, several of them in well under two seconds.

That is the same phenomenon as the macOS occurrences above, seen through a test that happens to have its own deadline. On the sharded macOS legs a Bun child stops making progress and the job simply goes quiet until the runner kills it; here a Bun child never starts producing at all and a 25-second wait reports it. In both cases the process exists, produces nothing, and does not exit.

Two consequences for how this should be investigated. First, any hypothesis scoped to macOS runner behaviour is now insufficient, because it has to explain Windows too. Second, the reproducible handles are better than they looked: the combo-failover hook timeout and this reset-credit case both catch the condition within 30 seconds, whereas the silent macOS form costs 16 to 18 minutes per observation.

PR #4957 does not cause this. It touches src/codex/history-provider.ts, src/codex/history-state-open.ts and structure/codex-home.md, and the new open policy is reachable only from preflightCodexHistoryInjection, which the reset-credit children never enter.

[lidge-jun]

Third distinct test, and it suggests the family is narrower than spawned child misbehaves.

Run 35286917745, windows 1/9, on branch codex/4897-teardown-owner-identity (PR #4960): management and data-plane credential separation > canceling an incomplete context body releases the actual listener turn failed at 15995.16ms. The other eight Windows shards in that run are green.

Put the three side by side:

• server combo failover 030 activation matrix > connect cancellation wins with 499, no backup, warning, or cooldown — hook timeout at ~30s, twice, on two unrelated branches.
• management and data-plane credential separation > canceling an incomplete context body releases the actual listener turn — ~16s, Windows.
• reset-credit auto-redeemer runtime (#822) > two processes reserve one durable id before either consume settles — 25s, Windows, two children with PIDs and no output.

Two of the three are explicitly about cancelling an in-flight request and observing that the thing underneath it is released. Both name the release in the test title, which means the assertion is precisely that a teardown completes. That is a much more specific hypothesis than the one I have been carrying: not that child processes misbehave generally, but that a cancellation can leave something attached that never settles, and whatever is waiting on it waits forever.

The reset-credit case may still belong to the same family or may not. Its children produce nothing at all rather than hanging mid-teardown, so it could be a second bug that happens to share a symptom. I would rather that stay an open question than be forced into one explanation.

Worth testing against the macOS occurrences: the 16-to-18-minute silences follow a passing test, so the relevant question is what the NEXT file's setup or the previous file's teardown was doing at that moment, not what the last passing test did.

[luvs01]

Scoped code-level finding: the connect-cancellation fixture cannot settle

I checked the current dev snapshot c9fd114cfac61570f249886a954800bd7dfa689e. There is a concrete fixture-lifecycle problem in the exact test named in the two macOS hook-timeout reports, independently of whether it explains the other silent-child occurrences.

Source at the reviewed revision:

const aStarted = deferred();
const a = serve(() => {
  aStarted.resolve();
  return new Promise<Response>(() => {});
});

The handler's promise has no resolver, abort handler, or test-owned release path. The test's finally restores console.warn only. The file-wide afterEach then awaits every server.stop(true) before flushResponseState() and the remaining cleanup. Thus this test deliberately leaves JavaScript handler work pending while teardown asks the server to finish.

Important boundary: this is a source-confirmed unresolved fixture, not yet a hosted macOS reproduction proving which teardown await stalls. It does not establish that the silent catalog/injection cases or the Windows reset-credit children have the same cause. Also, stop(true) is already present; recommending that flag alone is not a change to this code.

Narrow corrective direction

Replace only this fixture's never-settling handler with a test-owned deferred release:

const releaseA = deferred();
const a = serve(async () => {
  aStarted.resolve();
  await releaseA.promise;
  return new Response(null, { status: 204 });
});

Release it from the test's existing finally, after the cancellation assertions (also when an assertion throws), before global server teardown. Keep all existing assertions: 499 / client_cancelled, one cancellation receipt, no B request, no combo warning, and no cooldown. The release must not precede await pending, otherwise the fixture could accidentally make an uncancelled request complete and weaken the regression test. A bounded wait for fixture startup and explicit cleanup diagnostics would also make a failed setup distinguishable from a stop/flush hang; do not merely enlarge the 30-second hook budget.

Verification required before calling this a CI fix

1. At the pinned revision/Bun version, compare the original and releasable fixtures on hosted macOS, recording handler start/release and server.stop(true) start/completion separately from state flushing.
2. Run the named cancellation test and the complete tests/server/server-combo-failover-e2e.test.ts after the change, retaining every assertion above.
3. Re-run the normal macOS shards. Treat any remaining silent-child incident as still open, capturing previous-file teardown / next-file setup rather than blaming the last PASS line.

This comment does not claim a passing Bun/macOS rerun; none has been completed for this finding yet. #4905 remains the separate unsharded-control budget investigation and should not be closed by this fixture change.

[luvs01]

Implemented the scoped fixture correction from the earlier comment in #5014 (draft).

Current head: d2eb664118408d281843c111cfd0dcdf6313fcdb, including dev 50d72a1d09053be7be243bdaede3d2aac09ca830. The five-file diff replaces only the named test's never-settling handler with a finally-released gate, bounds startup/cancellation observation, retains every cancellation assertion, and adds five helper lifecycle cases. No production path, test-skip policy, hook timeout, or file-size limit is relaxed.

The helper passed standalone TypeScript and five Node contract checks, including a negative control. These are not Bun/macOS verification. Exact-head hosted Linux/macOS/Windows validation is queued as diagnostic run 35303541528 on the contributor fork's Actions; the normal PR workflow is also requested on the contributor fork's PR 545. No completed platform pass or full-suite result is claimed.

Please keep this issue open for the other silent-child occurrences, and keep #4905 separate. The PR addresses a concrete fixture-lifecycle defect, not a demonstrated common root cause for every report.

[lidge-jun]

A second test file shows the cold-spawn half of this, in a change that did not touch it.

Run on PR #5015, windows 2/9, job 105475642050: connected-client runtime probe scope > observes only the selected runtime and leaves full diagnostics available failed at 15339.29ms with expect(result.status).toBe(0) receiving null at tests/cli/cli-connect-readiness.test.ts:281. spawnSync returns a null status when the child is killed by its timeout rather than exiting, and that file passes timeout: INTERNAL_DEADLINE_MS, so a spawned child produced nothing for the full 15 seconds. 308 other cases in the same shard passed.

This is the same shape #4948 fixed in tests/cli/cli-status-json.test.ts: a spawnSync child whose cold module-graph startup is measured against a bound sized for warm children, where batch composition decides whether the file pays the cold cost. The remedy there was to move the cold start into an explicit setup outside the measured window rather than to raise the bound, and the same remedy is available here.

So the cold-spawn exposure is not specific to one file. Any test that spawns a real Bun child and measures it against a shared deadline carries it, and INTERNAL_DEADLINE_MS is read by more than twenty files. That is worth knowing before anyone reaches for the constant: raising it repeats the mistake tests/helpers/test-budget.ts already documents about SPAWN_BUDGET_MS.

Whether this is the same root cause as the silent 16-to-18-minute hangs or a separate cold-start problem that merely shares a symptom is still open. The distinction matters: a cold start is slow and finishes, a hang never does. These two files are slow-and-killed; the macOS shard silences and the reset-credit children are produce-nothing-and-never-exit.

[luvs01]

Hosted reproduction and native stack evidence

The original #5248 tested source tree was replayed with Bun 1.4.0 and shard 1/2, changing only diagnostics and the fork-only launch wrapper. Diagnostic run 35679762449, job 106593967562 (hosted on the contributor fork's Actions) again hit the unchanged 20-minute budget.

The phase trace establishes that case 1 finished database cleanup, environment restoration and directory removal. Case 2 then entered setup:catalog-probe:start and never returned. At 60 seconds of silence the owned Bun process had one defunct direct child. This rules out the preceding fixture cleanup as the stopping point in this reproduction.

The native sample's main thread resolves to spawnSync / spawn_maybe_sync / SpawnSyncEventLoop::tick_with_timeout. This is function-level symbolication from the official Bun 1.4.0 ARM64 profile archive: its UUID 4C4C440C-5555-3144-A11D-99D1CCBEAE57 matches the sampled executable, its SHA-256 matches the published digest, and each mapped address lies within the symbol's address/size interval. Other sampled threads were parked. The evidence is a parent synchronous-wait failure with an already exited child, not proof that the child never started.

At the exact Bun release source, a timeout kills the child and closes readable pipes but the loop continues while subprocess activity remains; subsequent ticks can have no deadline. An unreflected exit therefore explains why the configured probe timeout does not necessarily return control to the test runner. The capture does not yet identify why the exit was lost.

A controlled follow-up (diagnostic run 35681504245, job 106599252387, on the contributor fork's Actions) changed only the general suite argument from tests to ./tests, keeping source, test assertions, Bun version, shard ratio and budgets unchanged. It passed 15,113 general tests and 194 serial tests, zero failures, within ten minutes. Both runs selected 748/1496 general files; the 173 files through the original stall occur in identical order. The passing run covered all 748 general files and the four serial files owned by shard 1.

The trigger hypothesis is Bun's directory descriptor retention in filter-mode discovery: upstream fix #40016. That hypothesis must be distinguished from the captured wait location: no FD count was captured. The rooted-discovery correction is now in #5483 at abdccbe59d68ca5dd5632a04d6440a6cb6a443a3, with exact-argv and unchanged failure-propagation contracts. Both hosted shards are validating that exact checkout as diagnostic run 35682404784 on the contributor fork's Actions, while maintainer security review is requested. No merge or issue closure is claimed.

Final validation: shard 1 at corrected head cb41ba3b4c (diagnostic run 35683370022, job 106604940668, on the contributor fork's Actions) passed 14,722 tests including serial files, zero failures/errors. Shard 2 at abdccbe59d (run 35682404784, job 106601967172) passed 14,002. The only intervening edit corrects the shard-1-owned classifier test's old command anchor; shard 2 is unchanged. The initial final-head shard-1 run honestly failed on that missed anchor before it was corrected. No lane retry policy was added.

The Windows reports are treated separately in #5484: fixture-local LOCALAPPDATA isolation and disposable-child ACL helper seams. Hosted Windows at exact head 5f9d1046a8 (diagnostic run 35682566884, job 106602459110, on the contributor fork's Actions) passed 39 tests / 250 assertions. A reservation-ID reuse negative control failed as expected and original source bytes were restored before the positive run. The remaining incomplete-body cancellation case (run 35682759908, job 106603069723) passed unchanged in 1.13 seconds under the 15-second ceiling; this is current non-reproduction, not proof of its historical cause. The combo-cancellation fixture fix #5014 is already merged.

Local full prepush was attempted twice and still exceeded its 900-second Windows suite budget, with other Windows failures. Focused validations above are not a claim of full local CI success. Both corrective PRs await required checks and maintainer review/integration. Keep #4956 open until that integration is complete.