test(brain): canary 7 — local/Ollama parity, and the seam that makes it possible - #87
Merged
Conversation
…it possible The last of the seven release canaries, and the one that stayed unwritable longest. LocalBrain spawned a Python module with no injectable transport, so nothing in this repo could drive it: a search for `new LocalBrain` across test/ returned nothing, because it was not constructible under test. Adds a BrainSpawner seam. LocalBrain already speaks a newline-delimited JSON protocol over stdio, so a fake that speaks the same protocol is indistinguishable from the real child. Default behaviour is unchanged — the option simply defaults to node's spawn. What parity means here is deliberately NOT "identical events". Ollama is a pure-TypeScript loop; the Python path has a far richer vocabulary (stage, skill, telemetry, checkpoint). Requiring identical output would be a false contract. It means a host driving either brain sees the same SHAPE: the same tool calls in the same order, each carrying an id it can reply to, terminated by exactly one done. A host loop must not need to know which brain it is talking to. The test normalizes both transcripts to that view and requires them equal. Four assertions: both brains present the same shape to a host loop every tool_call from either brain carries a replyable id each brain terminates with exactly one done a brain that dies without a done still ends its stream The last one guards a hang rather than a wrong value: a host awaiting a terminal event that never arrives waits forever. Worth recording, because it nearly produced a false finding: the first run of that fourth test hung for ten minutes. The cause was the fake, not the product. LocalBrain ends its stream on the child's close event, and the fake's kill() closed stdout without emitting close — something a real child always does. A fake that is unfaithful about its termination signal turns a correct product into a hanging test. Fixed in the fake; the product path was right. Gates at this commit: npm run typecheck exit 0 npm test 1117 pass / 0 fail All seven canaries now run on every build: 1 denied mutation test/release_canaries.test.ts 2 live-child cancel test/process_tree.test.ts 3 reconnect replay test/release_canaries.test.ts 4 remote freshness test/worktree.test.ts 5 fake-gh ship test/ship_rail.test.ts 6 cap across reconnect test/release_canaries.test.ts 7 brain parity HERE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last of the seven release canaries, and the one that stayed unwritable
longest. LocalBrain spawned a Python module with no injectable transport, so
nothing in this repo could drive it: a search for
new LocalBrainacross test/returned nothing, because it was not constructible under test.
Adds a BrainSpawner seam. LocalBrain already speaks a newline-delimited JSON
protocol over stdio, so a fake that speaks the same protocol is indistinguishable
from the real child. Default behaviour is unchanged — the option simply defaults
to node's spawn.
What parity means here is deliberately NOT "identical events". Ollama is a
pure-TypeScript loop; the Python path has a far richer vocabulary (stage, skill,
telemetry, checkpoint). Requiring identical output would be a false contract.
It means a host driving either brain sees the same SHAPE: the same tool calls in
the same order, each carrying an id it can reply to, terminated by exactly one
done. A host loop must not need to know which brain it is talking to. The test
normalizes both transcripts to that view and requires them equal.
Four assertions:
both brains present the same shape to a host loop
every tool_call from either brain carries a replyable id
each brain terminates with exactly one done
a brain that dies without a done still ends its stream
The last one guards a hang rather than a wrong value: a host awaiting a terminal
event that never arrives waits forever.
Worth recording, because it nearly produced a false finding: the first run of
that fourth test hung for ten minutes. The cause was the fake, not the product.
LocalBrain ends its stream on the child's close event, and the fake's kill()
closed stdout without emitting close — something a real child always does. A
fake that is unfaithful about its termination signal turns a correct product
into a hanging test. Fixed in the fake; the product path was right.
Gates at this commit:
npm run typecheck exit 0
npm test 1117 pass / 0 fail
All seven canaries now run on every build:
1 denied mutation test/release_canaries.test.ts
2 live-child cancel test/process_tree.test.ts
3 reconnect replay test/release_canaries.test.ts
4 remote freshness test/worktree.test.ts
5 fake-gh ship test/ship_rail.test.ts
6 cap across reconnect test/release_canaries.test.ts
7 brain parity HERE