feat(claim-evidence): three grader guards, and a joinable mutation scope - #182
Conversation
…ng a claim for its environment Three rules in gradeFor decided a verdict the check had not earned. A check whose output says it never reached its input is now unrunnable whatever its exit status. The signature list ran only after a nonzero exit, but a shell pipeline exits with the status of its last stage, so `sha256sum <missing> | cut -d' ' -f1` printed the error and exited 0, and the error text was then compared to the expectation. Over 272 grade files, 29 of 61 recorded refutations were this shape. The list matches case-insensitively, only ever downgrades a refutation, and consults the expectation first so a claim that predicted the error is still refuted. A check killed at its deadline is unrunnable, decided before the expectation is compared. A deadline is a budget, not a verdict. A check whose text contains its own expected value is refused at both boundaries. It prints a value the claim's subject never had to produce. An occurrence glued to a word character does not count, so a short expectation does not refuse the check that reads it. The refusal for an expectation naming three or more key=value tokens is removed. It fired after the check exited 0 and above the comparison that would have passed: 173 of 1,001 claims carrying an expectation, 17.3%, were discarded that way, one of them matching byte for byte. gradeClaims grades one pass and flags a later claim that repeats an earlier claim's check, expectation, title and verdict. The verdict and the count do not change; N claims sharing one check are one verification counted N times, which a reader has to be able to see.
…the package withKnowledgeMutation is reentrant per async context, but the AsyncLocalStorage behind that was module-private. A consumer holding the store lock through its own wrapper could neither observe nor enter the scope, so calling any lock-taking function from this package inside its wrapper self-blocked against a lock the consumer already held. The only way out was to hold the lock by a private path and then forbid itself from calling this package while holding it, which is how a second lock over the same lockfile gets written. isKnowledgeMutationHeld(root) reports whether this async context already holds the root. runInKnowledgeMutationScope(root, hold, body) enters the scope on a lock the caller took by its own path: inside it, every lock-taking function in this package sees the root as held and runs inline. The owning path is now one function shared by a lock this package acquires and a lock the caller supplies, so the externally held lock gets the file transaction, the epoch, and the assertOwned checks at the same points rather than a weaker subset. The hold is required rather than assumed, because entering a scope for a lock the caller does not hold is the one way to defeat the single-writer rule. staleMs now documents what its window costs: the holder heartbeats at a third of it, so it also bounds how long a crashed holder wedges every other writer. Measured holds are 9 ms for one page, 586 ms for a 50-page batch, and about 390 ms for the largest store's longest promotion, against a 15-minute default.
Two exported shapes moved: CheckExecution gains optional timedOut and ClaimGrade gains optional duplicateOf. The surface record classifies a shape change as breaking whatever it did, so the boundary is major. A claim whose check contains its expected value is also refused where it was recorded before, at both the record and the grade boundary.
…enforced verifyGradeableEvidence bounds the check with timeoutMs, but a process killed at that bound reports no exit status of its own, so runBash returned 127 and the output was compared to the expectation like any other failure. The deadline rule in gradeFor could therefore never fire on this package's own execution path. runBash now sets CheckExecution.timedOut when its budget did the killing, and separates that from a caller's abort, which is a different verdict.
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 6d46826d
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-09-01T17:16:10Z
|
| State | Detail |
|---|---|
| Interrupted | webhook restarted |
No review verdict was produced for this run. Trigger a fresh review on the current PR head if the PR is still open.
tangletools · #182 · model: kimi-for-coding · updated 2026-09-01T17:17:32Z
Closes #179. Closes the first half of #180.
Every rule here was measured on live claims, and each one is carried today by a downstream copy that says it is mirroring this function. The copy exists because the capability was not here; this PR moves it here so the copy can go.
The three grader guards (#179)
A check that never reached its input is not a refutation.
UNRUNNABLE_SIGNATURESwas consulted only after a nonzero exit. A shell pipeline exits with the status of its last stage, sosha256sum <missing path> | cut -d' ' -f1prints "No such file or directory" and exits 0 — and the error text was then compared to the expectation and recorded as a refutation of research that never ran. Swept over 272 grade files: 29 of 61 recorded contradictions were this shape (7 a malformed check body, 7 a shell that could not parse the check, 6 a solver that could not open its problem file, 5Permission denied, 4cannot create directory). The list now reads the output whatever the exit status, matches case-insensitively, gained the five signatures above, only ever downgradescontradictedtounrunnable, and consults the expectation first so a claim that PREDICTED the error is still genuinely contradicted.A deadline is a budget, not a verdict. A check killed at its deadline grades
unrunnable, decided before the expectation is compared, so a slow solver that printed nothing is not recorded as refuted. Two ways to report one:CheckExecution.timedOutfor an executor that knows it did the killing, andDEADLINE_EXIT_CODE(124,timeout(1)'s status) for a grader holding only an exit status.A check may not carry its own answer. A check whose text contains its expected value prints a value the claim's subject never had to produce, so the comparison tests the author's typing rather than the artifact. Refused at both boundaries,
uncheckablewith a reason. One tightening over the downstream copy: an occurrence glued to a word character does not count, becauseexpect: 6appears insidenode report6.mjsand refusing a correct claim is the cost this lattice is calibrated to avoid.The fourth ask, the duplicate flag.
gradeClaims(claims)grades one pass and marks a later claim that repeats an earlier claim's check, expectation, normalized title and verdict withduplicateOf. Flagged, not dropped, and the verdict does not change. Only a full match counts, so one check shared by claims that say different things is one instrument used several times, not a duplicate.claimCheckKeyis exported for a grader that keeps its own map.And the half filed in the wrong repository (tangle-network/agent-runtime#1006): the refusal for an expectation carrying three or more
key=valuetokens is removed. It fired after the check exited 0 and immediately above the comparison that would have passed. The rule inverted its own intent —expect: OKis one token and is satisfied by any output containing those two letters, whileGRID OK cells=8 WIN=1 PARETO=6needs four independent numbers to coincide before it can pass falsely. Measured 2026-08-23: 173 of 1,001 rung-4-or-higher claims carrying an expectation, 17.3%, were discarded this way, including one whose output matched its expectation byte for byte. Multiline expectations are still refused, because the comparison is one contiguous substring.All four acceptance cases from #179 are tested: a zero-exit pipeline carrying an unreachable-input signature grades
unrunnable; the same case with an expectation naming the signature still gradescontradicted; a deadline-killed execution gradesunrunnablewith the deadline named; and a zero-exit check whose output contains a four-token expectation gradesverified.The mutation scope (#180, part 1)
withKnowledgeMutationis reentrant per async context, but theAsyncLocalStoragebehind that was module-private. A consumer holding the store lock through its own wrapper could neither observe nor enter the scope, so calling any lock-taking function from this package inside its wrapper self-blocked against a lock it already held. Its only option was to hold the lock by a private path and forbid itself from calling this package while holding it — which is how a second lock over the same lockfile gets written.isKnowledgeMutationHeld(root)reports whether this async context already holds the root.runInKnowledgeMutationScope(root, hold, body)enters the scope on a lock the caller took by its own path. Inside it, every lock-taking function in this package sees the root as held and runs inline.The owning path is now one function shared by a lock this package acquires and a lock the caller supplies, so an externally held lock gets the file transaction, the epoch, and the
assertOwned()checks at the same points rather than a weaker subset. The hold is required rather than assumed, because entering a scope for a lock the caller does not hold is the one way to defeat the single-writer rule.staleMsis now documented with its heartbeat relationship and the measured holds it should be sized against (9 ms for one page, 586 ms for a 50-page batch, ~390 ms for the largest store's longest promotion, against a 900 s default).The test takes the lock the way an external holder does, proves the self-block is real (
withKnowledgeMutationwithretries: 0rejects while the outside lock is held), then proves the same call runs inline inside the scope.What this lets the lab delete
discovery-lab/tools/oracle.mjs(701 lines):selfCertifying(:65), the threenormalized*helpers (:97-116),flagDuplicates(:175), thegradeExecutionwrapper (:235) andcontradictedButUnreachable(:321) — every one of them a mirror of whatgradeForandgradeClaimsnow do. The file keeps only the work that is genuinely the lab's:verifyIntegrityLedgers,priorVerifiedPool,gradeRun.discovery-lab/runner/grade.mjs: itsselfCertifying,unreachableInputandduplicateKeycopies, and the inlined signature list whose comment names this exact gap.discovery-lab/tools/kb-write.mjs(211 lines): the lock half — its ownAsyncLocalStorage(:109) and itsproper-lockfilecall over the same lockfile path (:111).withKnowledgeMutationalready takesstaleMsandretries, so withisKnowledgeMutationHeldthe wrapper reduces to a delegation.Not in this PR, and why
agent-evalalready contains the only correct detached process-group runner in the stack —SubprocessSandboxDriverinsrc/sandbox-harness.tsspawns detached, kills the negative pid on timeout, forces exit 124, and force-resolves so an orphaned grandchild cannot hang the caller — butsrc/index.tsexports only theSandboxDrivertype, so no consumer can reach it. Duplicating that logic here would give the stack two process runners. It is being extracted and exported there instead, with thebash -c 'sleep 60 & wait'descendant test fix(claim-evidence): verifyGradeableEvidence leaks descendant processes on timeout #181 asks for, after whichverifyGradeableEvidenceconsumes it and deletes its ownrunBash. What this PR does close is the reporting half:runBashnow setstimedOutwhen its own budget killed the check, so the deadline rule fires on this package's execution path instead of reading as exit 127.discovery-lab/tools/oracle-jail.mjstherefore stays until that lands.Verification
pnpm test: 838 passed, 12 skipped, 0 failed (850). Baseorigin/mainmeasured 807 passed with 1 failure and 12 skipped (820) — that failure,tests/memory/mem0.test.ts, is a load-sensitive hosted-index assert that passes on rerun and in isolation, and it passes on this branch. +30 tests.pnpm typecheck(bothsrcandcontractsprojects): clean.pnpm lint: clean.check:api-surface,check:path-containment,check:skills,check:version-bump: all pass.check:version-bump: two exported shapes moved (CheckExecutiongains optionaltimedOut,ClaimGradegains optionalduplicateOf), and the surface record classifies a shape change as breaking whatever it did. A claim whose check contains its expected value is also refused where it was recorded before, which is a real consumer-visible behavior change and is called out in the changelog.