ci: root macOS test discovery and diagnose silent process waits

[luvs01]

Summary

• Addresses the silent macOS shard failure in [Bug]: spawned Bun child processes stop producing output and never exit, on both macOS and Windows CI legs #4956. The general lane now uses ./tests, an explicit discovery root, instead of the repository-wide tests name filter. This keeps the same test inventory and shard ratio while avoiding unrelated directory traversal and descriptor retention in pinned Bun 1.4.0.
• Add a read-only observer beside the original foreground Bun/tee pipeline. After sustained log silence, it identifies the owning shell's single direct Bun child, records its descendant process names without argv/environment, and requests one native stack sample. It never signals or retries the suite. The original Bun exit status remains authoritative.
• Add CI-only phase boundaries to the catalog picker fixture and deterministic observer contracts for silence, continued progress, missing/ambiguous ownership, and termination while a diagnostic child is running. Exact argv and shell contracts preserve the full test root, shard ratio, exclusion ownership, first-failure propagation, budgets and assertions.
• The controlled reproduction links the macOS stall to Bun's synchronous child wait: an already exited child remains unreaped while the parent spins in SpawnSyncEventLoop::tick_with_timeout. Rooted discovery removes the observed failure trigger; this PR does not patch Bun itself or assert that the separately reported Windows fixture failures share that root cause. Maintainer sponsorship and security review remain required before merge.

Verification

• Bun 1.4.0: bun test tests/codex-integration/catalog-full-picker-order.test.ts tests/ci-workflows/macos-serial-lanes.test.ts tests/ci-workflows/ci-workflows.test.ts --timeout 60000 — 170 pass, 29 skip, 0 fail; Unix-only shell cases retain their existing Windows skip.
• Executed tests/fixtures/macos-stall-observer.sh through Git Bash — all five observer scenarios passed, including reaping the diagnostic sleep while leaving the owner alive.
• bun run typecheck, bun run structure:check, bun run privacy:scan, shell syntax checks and git diff --check passed.
• bun run prepush was run against the actual dev comparison and failed: its test subprocess reached the 15-minute budget (exit 124), with Windows ACL/EBUSY/timeouts among the failures. This is not a full-suite pass.
• Hosted baseline reproduced the original fix(kiro): avoid mixed-script estimator allocations #5248 tree under Bun 1.4.0 and shard 1/2: first-case cleanup completed, second-case catalog probing stalled, and the job reached 20 minutes. Native frames were symbolicated against the matching official ARM64 binary UUID to spawnSync / SpawnSyncEventLoop::tick_with_timeout.
• The controlled follow-up changed only tests to ./tests. It passed 15,113 general tests and 194 serial tests, with zero failures, in under ten minutes. Both executions selected 748/1496 general files; all 173 file names through the original stopping point appear in identical order. The completed run then covered all 748 general files and four owned serial files.
• After integration, focused workflow contracts passed 138 tests, 29 existing platform skips, zero failures; structure checks passed. The full prepush was rerun after the workflow edit and again exhausted its 900-second suite budget on Windows (exit 124), with unrelated Windows failures; it is not green.
• At abdccbe59d, both hosted macOS shards finished without the silent hang. Shard 2 passed 13,958 general tests plus 44 serial tests. Shard 1 completed 14,523 general tests, zero assertion failures, but correctly exited 1 on one module-load error: the crash-classifier contract still searched for the old bare-root command. cb41ba3b4c corrects that overlooked anchor; its seven focused tests / 52 assertions pass.
• Corrected shard 1 at exact head cb41ba3b4c (diagnostic run 35683370022, job 106604940668, contributor fork Actions) passed 14,528 general tests plus 194 serial tests, zero failures/errors. Shard 2 at abdccbe59d (diagnostic run 35682404784, job 106601967172, contributor fork Actions) is unchanged by the final one-line edit in the shard-1-owned classifier file; its 14,002 passes remain applicable. No tests were removed, no timeout was increased, and no failed attempt was converted to green by a workflow retry.

Evidence

• Failing baseline capture (diagnostic run 35679762449, job 106593967562, contributor fork Actions) and passing controlled follow-up (diagnostic run 35681504245, job 106599252387, contributor fork Actions).
• Pinned Bun synchronous wait and upstream directory-descriptor retention fix. The native capture proves the wait location; descriptor pressure is the source-supported trigger hypothesis, not a captured FD count.

Checklist

• Scope stays focused and avoids unrelated cleanup.
• Docs or release notes were updated when needed.
• Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. Independent local review found no remaining blocking issue; explicit maintainer review is still pending.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ Deterministic PR hygiene checks passed.

[lidge-jun]

리뷰 · 우선순위 54 / 80

이 PR은 #4956에서 보이는 macOS CI 멈춤을 “고치는” 패치가 아닙니다. 호스트 macOS shard가 어떤 테스트를 통과한 뒤 출력을 멈추고 20분 벽에서 잘리는 문제를 나중에 볼 수 있게, 옆에 읽기 전용 관찰자만 붙입니다. run_macos_suite가 Bun 테스트를 돌리는 동안 scripts/ci/sample-macos-stall.sh를 백그라운드로 켜 두고, 로그가 약 60초 동안 안 늘면 소유 셸의 직계 Bun 자식이 하나인지 확인한 뒤, 그 자손 프로세스 이름만 찍고 sample로 스택을 한 번 뜹니다. argv·환경 변수는 안 찍고, 스위트에 시그널을 보내거나 재시도하지 않으며, 종료 코드도 원래 Bun 결과가 그대로입니다. 카탈로그 picker 픽스처에는 CI에서만 보이는 setup/writer/cleanup 단계 로그를 넣었고, observer 동작은 mock ps/sample/sleep 픽스처로 다섯 시나리오를 검증합니다. draft이고, 호스트 macOS에서 실제 스택 수집은 아직 안 했고, 작성자도 보안·스폰서 리뷰가 필요하다고 적어 두었습니다.

라인 - scripts/ci/sample-macos-stall.sh의 sample 출력은 argv를 빼도 스택 프레임·심볼 안에 시크릿·경로·요청 조각이 섞일 수 있습니다. 프로세스 목록은 comm만 쓰도록 잘 막았지만, 스택 덤프 쪽은 아직 메인테이너 보안 판단이 필요합니다.
라인 - 관찰자는 침묵을 한 번 잡은 뒤 샘플하고 바로 끝납니다. 같은 suite 안에서 잠깐 출력이 나왔다가 다시 멈추면 두 번째 멈춤은 안 잡힙니다. #4956처럼 “한 번 PASS 후 긴 침묵”이면 맞지만, 중간에 로그가 튀는 형태면 증거가 비일 수 있습니다.
라인 - 로컬·픽스처 검증은 있으나, 호스트 macOS runner에서 실제 sample·ps 경로가 기대한 증거를 남기는지는 PR 본문대로 아직 미검증입니다. 픽스처의 fake sample/ps만으로는 CI 실측을 대신할 수 없습니다.
라인 - PR이 draft이고 보안 체크리스트가 비어 있습니다. 진단용이라 런타임 제품 경로는 안 건드리지만, CI 로그에 네이티브 스택이 올라가는 변경이라 “준비됨” 표시 전에 한 번 더 보는 편이 안전합니다.

메인테이너의 판단이 필요한 지점

• 스택 샘플을 CI 로그에 그대로 올려도 되는지, 아니면 redaction·길이 제한·그룹 접기 이상(또는 아티팩트만)이 필요한지.
• draft·보안 리뷰를 끝낸 뒤 머지할지, 아니면 호스트 macOS에서 실제 한 번 침묵→샘플이 찍힌 run 증거를 먼저 볼지.
• 한 번만 샘플하고 끝나는 설계를 [Bug]: spawned Bun child processes stop producing output and never exit, on both macOS and Windows CI legs #4956 현재 패턴에 충분한지, 아니면 suite 종료까지 재무장할지.

너의 추천

• 방향은 맞고, 스위트 exit를 건드리지 않는 읽기 전용 진단 전제조건으로 잘 짜여 있습니다. 지금은 draft·보안·호스트 실측이 열려 있으니 바로 머지보다, 보안 리뷰와 호스트 macOS에서 샘플이 실제로 남는 한 run을 본 뒤 Ready로 올리는 쪽을 권합니다. types/config 중복 PR 이슈는 이 변경과 무관합니다.

이 댓글은 grok-bot이 작성했습니다

[lidge-jun]

discord 좀 들어와주세용

[lidge-jun]

Thanks @luvs01. Part of this PR landed on dev through #5509 (squash commit 798c07c), with a Co-authored-by trailer crediting you.

What landed:

• scripts/ci/sample-macos-stall.sh as a standalone diagnostic helper. The bundle also added prefix redaction and a byte cap to its emitted output.
• The observer regression: tests/fixtures/macos-stall-observer.sh and its coverage in tests/ci-workflows/macos-serial-lanes.test.ts.
• The CI-only phase boundaries in tests/codex-integration/catalog-full-picker-order.test.ts (unchanged from this branch).
• The matching note in structure/ops/docs-and-release.md.

What did not land: the .github/workflows/ci.yml wiring that started the observer inside run_macos_suite, and the ./tests discovery root for the macOS shard invocation, along with the two assertion updates that pinned them. Current dev replaced that long-lived shard function with bounded fresh-process batches, so neither change applies to the current workflow, and the helper is intentionally left unwired.

Wiring stall observation into the batch runner is still open. It would need a new change against the current macOS batch steps, so this PR stays open for that discussion.

[lidge-jun]

Thank you, @luvs01. This PR is superseded by the lane bundle #5597, and its content is already on dev:

• 798c07c7a4 (chore(ci): consolidate development hooks and hosted reliability fixes #5509) carried the stall-observer helper (with output redaction added), the observer fixture and regression, the catalog picker phase markers, and the structure note.
• The remaining hunks here (rooted ./tests discovery, observer wiring, and the matching anchor edits) target the long-lived run_macos_suite step, which 2dec4be4fd (ci: bound ordinary macOS shard process lifetimes #5520) replaced with scripts/ci/run-bun-test-batches.sh. That runner passes explicit file paths and bounds every Bun process with a timeout, so those hunks no longer have a target.

#5597 records this disposition (ALREADY ON DEV). Closing as superseded.