Skip to content

fix(runtime): async tool execution with process-tree teardown - #84

Merged
AetherAI3 merged 1 commit into
mainfrom
fix/async-tool-runtime
Aug 19, 2026
Merged

fix(runtime): async tool execution with process-tree teardown#84
AetherAI3 merged 1 commit into
mainfrom
fix/async-tool-runtime

Conversation

@AetherAI3

Copy link
Copy Markdown
Owner

Problem

run_shell and run_tests used spawnSync with a timeout. Two defects the caller could not see:

1. A timeout signalled the shell, not the work. spawnSync's timeout reaches the direct child — which is cmd.exe or sh. Whatever the user actually started (npm test, pytest, a compiler) was orphaned and kept running, holding ports, files and CPU, while the call returned looking like a clean timeout. On Windows the cmd.exe shell made it near-certain.

2. The event loop was blocked for up to 15 minutes, freezing heartbeats, the renderer, and any AbortController. That is why Ctrl+C could not interrupt a long test run: nothing was listening.

Contract

POSIX spawn into a new process group; kill the group; escalate SIGTERM → SIGKILL after 2s so a runner that traps SIGTERM cannot outlive its timeout
Windows no equivalent grouping, so taskkill /T /F walks the tree
timeout exit 124
cancellation exit 130 — one is the clock, one is the operator, and the caller needs to tell them apart

Output is drained past the retention cap rather than truncated at the pipe, so a large-output command can never block on a full buffer. Results settle on close rather than exit, so a test summary arriving with the exit is not lost.

The consequential part: verify_gate

finalVerify is the gate that stops a brain claiming success on a red tree — and it called the synchronous path.

Left alone, production would have received [tool run_tests is async] while its tests kept passing against a fake. A silent break with green CI.

So VerifyRunner now names executeAsync rather than execute:

export interface VerifyRunner {
  executeAsync(name: string, args: Record<string, unknown>): Promise<ToolResult>;
}

That is deliberate. A fake still implementing the sync method fails to compile instead of quietly diverging from what production calls. All 14 verify_gate tests were updated by adding await and renaming the fake's method — no assertion changed.

Tests

4 new, asserting survival by PID rather than a prompt return. A parent that exits while its grandchild keeps running looks identical to success from the caller's side — which is precisely how this went unnoticed.

test proves
a timed-out command kills its whole tree the grandchild pid is gone, not just the shell
an aborted command kills its tree and reports aborted 130 ≠ 124, and no survivors
a normal command still returns its real exit code exit 3 stays exit 3
the event loop keeps running while a command is in flight a timer ticks during the call

That last one distinguishes async execution from a merely faster synchronous one — it fails if no timer can tick.

Mutation-checked on the invariant that mattered. Letting done.ok upgrade a red run:

result
mutated 12 pass / 2 failbrain done.ok=true while host tests are RED → never ok, and the 24-bug corpus test
restored 14 / 14

The ground-truth gate survived the conversion. That was the whole risk of this change.

command result
npm run typecheck exit 0
npm test 1094 pass / 0 fail

Known limits

  • git_commit still uses spawnSync (git_commit_guard.ts), with no timeout at all. Left alone here: it is a short-lived, non-interactive git invocation, and folding it in would have widened a change that touches the verification gate. Worth its own pass.
  • hostLoop still has no turn or wall-clock budget for LocalBrain/CloudBrain; only OllamaBrain self-limits.
  • chat.ts still re-implements the host loop separately from code.ts, and the REPL path still has no finalVerify at all. That is the shared-runtime lane, not this one.

Scope

src/core/tool_executor.ts, src/core/verify_gate.ts, src/commands/code.ts, plus test updates in verify_gate, tool_executor and bridge. New: test/process_tree.test.ts.

run_shell and run_tests used spawnSync with a timeout. Two defects the caller
could not see:

  * a timeout signals the DIRECT child, which is the shell. Whatever the user
    actually started — npm test, pytest, a compiler — was orphaned and kept
    running, holding ports, files and CPU, while the call returned looking like
    a clean timeout. On Windows the cmd.exe shell made it near-certain.
  * the event loop was blocked for the duration (up to 15 minutes), freezing
    heartbeats, the renderer and any AbortController. That is why Ctrl+C could
    not interrupt a long test run: nothing was listening.

`run` is now asynchronous and tree-aware. POSIX spawns into a new process group
and kills the group, escalating SIGTERM to SIGKILL after 2s so a runner that
traps SIGTERM cannot outlive its timeout. Windows has no equivalent grouping,
so `taskkill /T /F` walks the tree instead. Timeout and cancellation resolve
distinctly — 124 and 130 — because one is the clock and one is the operator,
and the caller needs to tell them apart.

Output is drained past the retention cap rather than truncated at the pipe, so
a large-output command can never block on a full buffer, and results settle on
'close' rather than 'exit' so a test summary arriving with the exit is not lost.

run_shell and run_tests move from the synchronous execute() to executeAsync,
alongside the web tools that were already there.

The consequential part is verify_gate. finalVerify is the gate that stops a
brain claiming success on a red tree, and it called the synchronous path.
Left alone it would have received "[tool run_tests is async]" in production
while its tests kept passing against a fake — a silent break with green CI.

So VerifyRunner now names executeAsync rather than execute. That is deliberate:
a fake still implementing the sync method fails to compile instead of quietly
diverging from what production calls. All 14 verify_gate tests were updated by
adding await and renaming the fake's method — no assertion changed.

Tests: 4 new, asserting survival by PID rather than a prompt return. A parent
that exits while its grandchild keeps running looks identical to success from
the caller's side, which is how this went unnoticed.

  a timed-out command kills its whole tree, not just the shell
  an aborted command kills its tree and reports aborted, not timed out
  a normal command still returns its real exit code and output
  the event loop keeps running while a command is in flight

The last one distinguishes async execution from a merely faster synchronous
one: it fails if a timer cannot tick during the call.

Mutation-checked on the invariant that mattered: letting done.ok upgrade a red
run fails "brain done.ok=true while host tests are RED → never ok" and the
24-bug corpus test (12 pass / 2 fail). Restored, 14 / 14. The ground-truth gate
survived the conversion.

Gates at this commit:
  npm run typecheck   exit 0
  npm test            1094 pass / 0 fail
@AetherAI3
AetherAI3 merged commit a6be323 into main Aug 19, 2026
5 checks passed
@AetherAI3
AetherAI3 deleted the fix/async-tool-runtime branch August 19, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant