docs: synchronize recent audit findings - #98
Conversation
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
|
|
||
| ```ts | ||
| run.seamOutput; // what the seam call returned | ||
| run.seamUsage; // token usage reported by the seam call, when available |
There was a problem hiding this comment.
🟡 Docs describe a seam-result token-usage field that the tool never returns
The seam-run result is documented (run.seamUsage at docs/evals.md:188) as carrying the seam call's token usage, but the actual result value has no such property, so anyone following the docs reads back nothing.
Impact: Developers who read the seam's token usage from this documented field get undefined and their usage/scoring code silently misbehaves.
Field is absent from the RunSeamResult contract
The result type RunSeamResult in src/seam.ts:141-158 declares only result, seamOutput, callsBeforeSeam, before, and after. There is no seamUsage field, and runSeam (src/seam.ts:381-387) never returns one. The docs snippet checker does not catch this because the snippet's run binding resolves to the ambient declare const run: any in docs/snippet-globals.ts:114, so the property access is not type-checked.
Prompt for agents
The docs at docs/evals.md:188 document run.seamUsage, but the RunSeamResult interface in src/seam.ts (lines 141-158) has no seamUsage field and runSeam never returns one. Either remove this documented field from the doc, or (if the intent of this docs-sync PR is that the field should exist) add a seamUsage field to RunSeamResult and populate it in runSeam. Confirm which direction is correct against the current source and make the docs match reality.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Valid; fixed in 215bede by removing the nonexistent run.seamUsage field from the documented result contract. Full docs check (115 snippets), source typecheck, and focused seam/verify tests (48) pass.
| run.seamOutput; // what the seam call returned | ||
| run.seamUsage; // token usage reported by the seam call, when available | ||
| run.callsBeforeSeam; // model calls made before it (-1 if never reached) | ||
| run.calls; // ordered text-call ledger: routing key, source, and seam marker |
There was a problem hiding this comment.
🟡 Docs describe a seam-result call ledger field that the tool never returns
The seam-run result is documented (run.calls at docs/evals.md:190) as exposing an ordered ledger of the run's text calls, but the actual result value has no such property, so anyone following the docs reads back nothing.
Impact: Developers who inspect the documented per-call ledger get undefined and any code iterating it breaks.
Field is absent from the RunSeamResult contract
The result type RunSeamResult in src/seam.ts:141-158 declares only result, seamOutput, callsBeforeSeam, before, and after. There is no calls field; calls inside runSeam (src/seam.ts:226) is only a local counter and is never returned. The docs snippet checker misses this because the snippet's run binding resolves to the ambient declare const run: any in docs/snippet-globals.ts:114, so the property access is not type-checked.
Prompt for agents
The docs at docs/evals.md:190 document run.calls as an ordered text-call ledger, but the RunSeamResult interface in src/seam.ts (lines 141-158) has no calls field and runSeam never returns one (calls at src/seam.ts:226 is only a local counter). Either remove this documented field, or (if this docs-sync PR intends it to exist) add a calls ledger field to RunSeamResult and populate it in runSeam. Verify against the source and make the docs match reality.
Was this helpful? React with 👍 or 👎 to provide feedback.
| `canReach(machine, target, opts)` wraps `explorePaths` to answer whether a state | ||
| path or snapshot predicate is reachable, with a witness path when it is. |
There was a problem hiding this comment.
🟡 Docs claim a reachability check accepts a predicate it does not support
The reachability helper is documented (canReach at docs/verify.md:168-169) as answering whether a state path OR a snapshot predicate is reachable, but the function only accepts a state-path string, so the promised predicate form does not work.
Impact: A reader who passes a predicate function as documented hits a broken call instead of the described behavior.
canReach signature only accepts a string state path
canReach in src/verify.ts:1214-1226 is typed as canReach(machine, statePath: string, options) and internally matches with snapshot.matches(statePath as never). There is no overload or code path accepting a snapshot predicate function; passing one would not match anything. The doc text 'a state path or snapshot predicate is reachable' overstates the contract.
| `canReach(machine, target, opts)` wraps `explorePaths` to answer whether a state | |
| path or snapshot predicate is reachable, with a witness path when it is. | |
| `canReach(machine, target, opts)` wraps `explorePaths` to answer whether a state | |
| path is reachable, with a witness path when it is. |
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
Validation
pnpm docs:check(115 blocks checked, 0 failed)git diff --check