fix(runtime): async tool execution with process-tree teardown - #84
Merged
Conversation
run_shell and run_tests used spawnSync with a timeout. Two defects the caller
could not see:
* a timeout signals the DIRECT child, which is the shell. Whatever the user
actually started — npm test, pytest, a compiler — was orphaned and kept
running, holding ports, files and CPU, while the call returned looking like
a clean timeout. On Windows the cmd.exe shell made it near-certain.
* the event loop was blocked for the duration (up to 15 minutes), freezing
heartbeats, the renderer and any AbortController. That is why Ctrl+C could
not interrupt a long test run: nothing was listening.
`run` is now asynchronous and tree-aware. POSIX spawns into a new process group
and kills the group, escalating SIGTERM to SIGKILL after 2s so a runner that
traps SIGTERM cannot outlive its timeout. Windows has no equivalent grouping,
so `taskkill /T /F` walks the tree instead. Timeout and cancellation resolve
distinctly — 124 and 130 — because one is the clock and one is the operator,
and the caller needs to tell them apart.
Output is drained past the retention cap rather than truncated at the pipe, so
a large-output command can never block on a full buffer, and results settle on
'close' rather than 'exit' so a test summary arriving with the exit is not lost.
run_shell and run_tests move from the synchronous execute() to executeAsync,
alongside the web tools that were already there.
The consequential part is verify_gate. finalVerify is the gate that stops a
brain claiming success on a red tree, and it called the synchronous path.
Left alone it would have received "[tool run_tests is async]" in production
while its tests kept passing against a fake — a silent break with green CI.
So VerifyRunner now names executeAsync rather than execute. That is deliberate:
a fake still implementing the sync method fails to compile instead of quietly
diverging from what production calls. All 14 verify_gate tests were updated by
adding await and renaming the fake's method — no assertion changed.
Tests: 4 new, asserting survival by PID rather than a prompt return. A parent
that exits while its grandchild keeps running looks identical to success from
the caller's side, which is how this went unnoticed.
a timed-out command kills its whole tree, not just the shell
an aborted command kills its tree and reports aborted, not timed out
a normal command still returns its real exit code and output
the event loop keeps running while a command is in flight
The last one distinguishes async execution from a merely faster synchronous
one: it fails if a timer cannot tick during the call.
Mutation-checked on the invariant that mattered: letting done.ok upgrade a red
run fails "brain done.ok=true while host tests are RED → never ok" and the
24-bug corpus test (12 pass / 2 fail). Restored, 14 / 14. The ground-truth gate
survived the conversion.
Gates at this commit:
npm run typecheck exit 0
npm test 1094 pass / 0 fail
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
run_shellandrun_testsusedspawnSyncwith atimeout. Two defects the caller could not see:1. A timeout signalled the shell, not the work.
spawnSync's timeout reaches the direct child — which iscmd.exeorsh. Whatever the user actually started (npm test,pytest, a compiler) was orphaned and kept running, holding ports, files and CPU, while the call returned looking like a clean timeout. On Windows thecmd.exeshell made it near-certain.2. The event loop was blocked for up to 15 minutes, freezing heartbeats, the renderer, and any
AbortController. That is why Ctrl+C could not interrupt a long test run: nothing was listening.Contract
taskkill /T /Fwalks the treeOutput is drained past the retention cap rather than truncated at the pipe, so a large-output command can never block on a full buffer. Results settle on
closerather thanexit, so a test summary arriving with the exit is not lost.The consequential part:
verify_gatefinalVerifyis the gate that stops a brain claiming success on a red tree — and it called the synchronous path.Left alone, production would have received
[tool run_tests is async]while its tests kept passing against a fake. A silent break with green CI.So
VerifyRunnernow namesexecuteAsyncrather thanexecute:That is deliberate. A fake still implementing the sync method fails to compile instead of quietly diverging from what production calls. All 14
verify_gatetests were updated by addingawaitand renaming the fake's method — no assertion changed.Tests
4 new, asserting survival by PID rather than a prompt return. A parent that exits while its grandchild keeps running looks identical to success from the caller's side — which is precisely how this went unnoticed.
That last one distinguishes async execution from a merely faster synchronous one — it fails if no timer can tick.
Mutation-checked on the invariant that mattered. Letting
done.okupgrade a red run:brain done.ok=true while host tests are RED → never ok, and the 24-bug corpus testThe ground-truth gate survived the conversion. That was the whole risk of this change.
npm run typechecknpm testKnown limits
git_commitstill usesspawnSync(git_commit_guard.ts), with no timeout at all. Left alone here: it is a short-lived, non-interactive git invocation, and folding it in would have widened a change that touches the verification gate. Worth its own pass.hostLoopstill has no turn or wall-clock budget forLocalBrain/CloudBrain; onlyOllamaBrainself-limits.chat.tsstill re-implements the host loop separately fromcode.ts, and the REPL path still has nofinalVerifyat all. That is the shared-runtime lane, not this one.Scope
src/core/tool_executor.ts,src/core/verify_gate.ts,src/commands/code.ts, plus test updates inverify_gate,tool_executorandbridge. New:test/process_tree.test.ts.