Skip to content

feat(claim-evidence): three grader guards, and a joinable mutation scope - #182

Merged
drewstone merged 4 commits into
mainfrom
feat/grader-guards-and-detached-checks
Sep 1, 2026
Merged

feat(claim-evidence): three grader guards, and a joinable mutation scope#182
drewstone merged 4 commits into
mainfrom
feat/grader-guards-and-detached-checks

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Closes #179. Closes the first half of #180.

Every rule here was measured on live claims, and each one is carried today by a downstream copy that says it is mirroring this function. The copy exists because the capability was not here; this PR moves it here so the copy can go.

The three grader guards (#179)

A check that never reached its input is not a refutation. UNRUNNABLE_SIGNATURES was consulted only after a nonzero exit. A shell pipeline exits with the status of its last stage, so sha256sum <missing path> | cut -d' ' -f1 prints "No such file or directory" and exits 0 — and the error text was then compared to the expectation and recorded as a refutation of research that never ran. Swept over 272 grade files: 29 of 61 recorded contradictions were this shape (7 a malformed check body, 7 a shell that could not parse the check, 6 a solver that could not open its problem file, 5 Permission denied, 4 cannot create directory). The list now reads the output whatever the exit status, matches case-insensitively, gained the five signatures above, only ever downgrades contradicted to unrunnable, and consults the expectation first so a claim that PREDICTED the error is still genuinely contradicted.

A deadline is a budget, not a verdict. A check killed at its deadline grades unrunnable, decided before the expectation is compared, so a slow solver that printed nothing is not recorded as refuted. Two ways to report one: CheckExecution.timedOut for an executor that knows it did the killing, and DEADLINE_EXIT_CODE (124, timeout(1)'s status) for a grader holding only an exit status.

A check may not carry its own answer. A check whose text contains its expected value prints a value the claim's subject never had to produce, so the comparison tests the author's typing rather than the artifact. Refused at both boundaries, uncheckable with a reason. One tightening over the downstream copy: an occurrence glued to a word character does not count, because expect: 6 appears inside node report6.mjs and refusing a correct claim is the cost this lattice is calibrated to avoid.

The fourth ask, the duplicate flag. gradeClaims(claims) grades one pass and marks a later claim that repeats an earlier claim's check, expectation, normalized title and verdict with duplicateOf. Flagged, not dropped, and the verdict does not change. Only a full match counts, so one check shared by claims that say different things is one instrument used several times, not a duplicate. claimCheckKey is exported for a grader that keeps its own map.

And the half filed in the wrong repository (tangle-network/agent-runtime#1006): the refusal for an expectation carrying three or more key=value tokens is removed. It fired after the check exited 0 and immediately above the comparison that would have passed. The rule inverted its own intent — expect: OK is one token and is satisfied by any output containing those two letters, while GRID OK cells=8 WIN=1 PARETO=6 needs four independent numbers to coincide before it can pass falsely. Measured 2026-08-23: 173 of 1,001 rung-4-or-higher claims carrying an expectation, 17.3%, were discarded this way, including one whose output matched its expectation byte for byte. Multiline expectations are still refused, because the comparison is one contiguous substring.

All four acceptance cases from #179 are tested: a zero-exit pipeline carrying an unreachable-input signature grades unrunnable; the same case with an expectation naming the signature still grades contradicted; a deadline-killed execution grades unrunnable with the deadline named; and a zero-exit check whose output contains a four-token expectation grades verified.

The mutation scope (#180, part 1)

withKnowledgeMutation is reentrant per async context, but the AsyncLocalStorage behind that was module-private. A consumer holding the store lock through its own wrapper could neither observe nor enter the scope, so calling any lock-taking function from this package inside its wrapper self-blocked against a lock it already held. Its only option was to hold the lock by a private path and forbid itself from calling this package while holding it — which is how a second lock over the same lockfile gets written.

  • isKnowledgeMutationHeld(root) reports whether this async context already holds the root.
  • runInKnowledgeMutationScope(root, hold, body) enters the scope on a lock the caller took by its own path. Inside it, every lock-taking function in this package sees the root as held and runs inline.

The owning path is now one function shared by a lock this package acquires and a lock the caller supplies, so an externally held lock gets the file transaction, the epoch, and the assertOwned() checks at the same points rather than a weaker subset. The hold is required rather than assumed, because entering a scope for a lock the caller does not hold is the one way to defeat the single-writer rule. staleMs is now documented with its heartbeat relationship and the measured holds it should be sized against (9 ms for one page, 586 ms for a 50-page batch, ~390 ms for the largest store's longest promotion, against a 900 s default).

The test takes the lock the way an external holder does, proves the self-block is real (withKnowledgeMutation with retries: 0 rejects while the outside lock is held), then proves the same call runs inline inside the scope.

What this lets the lab delete

  • discovery-lab/tools/oracle.mjs (701 lines): selfCertifying (:65), the three normalized* helpers (:97-116), flagDuplicates (:175), the gradeExecution wrapper (:235) and contradictedButUnreachable (:321) — every one of them a mirror of what gradeFor and gradeClaims now do. The file keeps only the work that is genuinely the lab's: verifyIntegrityLedgers, priorVerifiedPool, gradeRun.
  • discovery-lab/runner/grade.mjs: its selfCertifying, unreachableInput and duplicateKey copies, and the inlined signature list whose comment names this exact gap.
  • discovery-lab/tools/kb-write.mjs (211 lines): the lock half — its own AsyncLocalStorage (:109) and its proper-lockfile call over the same lockfile path (:111). withKnowledgeMutation already takes staleMs and retries, so with isKnowledgeMutationHeld the wrapper reduces to a delegation.

Not in this PR, and why

  • fix(claim-evidence): verifyGradeableEvidence leaks descendant processes on timeout #181, the descendant leak. The fix does not belong in this package. agent-eval already contains the only correct detached process-group runner in the stack — SubprocessSandboxDriver in src/sandbox-harness.ts spawns detached, kills the negative pid on timeout, forces exit 124, and force-resolves so an orphaned grandchild cannot hang the caller — but src/index.ts exports only the SandboxDriver type, so no consumer can reach it. Duplicating that logic here would give the stack two process runners. It is being extracted and exported there instead, with the bash -c 'sleep 60 & wait' descendant test fix(claim-evidence): verifyGradeableEvidence leaks descendant processes on timeout #181 asks for, after which verifyGradeableEvidence consumes it and deletes its own runBash. What this PR does close is the reporting half: runBash now sets timedOut when its own budget killed the check, so the deadline rule fires on this package's execution path instead of reading as exit 127. discovery-lab/tools/oracle-jail.mjs therefore stays until that lands.
  • feat(mutation-lock): expose the reentrancy scope, and add a null control over verified evidence #180 part 2, the null control. Untouched. It needs check-text and named-script path rewriting against an empty workspace, which is a feature rather than a guard, and the measurement behind it (100 of 102 apparent evidence-independent passes were an artifact of a naive null; the genuine count was 1 of 300) argues for building it carefully rather than alongside this.

Verification

  • pnpm test: 838 passed, 12 skipped, 0 failed (850). Base origin/main measured 807 passed with 1 failure and 12 skipped (820) — that failure, tests/memory/mem0.test.ts, is a load-sensitive hosted-index assert that passes on rerun and in isolation, and it passes on this branch. +30 tests.
  • pnpm typecheck (both src and contracts projects): clean. pnpm lint: clean. check:api-surface, check:path-containment, check:skills, check:version-bump: all pass.
  • Major bump to 12.0.0, demanded by check:version-bump: two exported shapes moved (CheckExecution gains optional timedOut, ClaimGrade gains optional duplicateOf), and the surface record classifies a shape change as breaking whatever it did. A claim whose check contains its expected value is also refused where it was recorded before, which is a real consumer-visible behavior change and is called out in the changelog.

…ng a claim for its environment

Three rules in gradeFor decided a verdict the check had not earned.

A check whose output says it never reached its input is now unrunnable
whatever its exit status. The signature list ran only after a nonzero
exit, but a shell pipeline exits with the status of its last stage, so
`sha256sum <missing> | cut -d' ' -f1` printed the error and exited 0,
and the error text was then compared to the expectation. Over 272 grade
files, 29 of 61 recorded refutations were this shape. The list matches
case-insensitively, only ever downgrades a refutation, and consults the
expectation first so a claim that predicted the error is still refuted.

A check killed at its deadline is unrunnable, decided before the
expectation is compared. A deadline is a budget, not a verdict.

A check whose text contains its own expected value is refused at both
boundaries. It prints a value the claim's subject never had to produce.
An occurrence glued to a word character does not count, so a short
expectation does not refuse the check that reads it.

The refusal for an expectation naming three or more key=value tokens is
removed. It fired after the check exited 0 and above the comparison
that would have passed: 173 of 1,001 claims carrying an expectation,
17.3%, were discarded that way, one of them matching byte for byte.

gradeClaims grades one pass and flags a later claim that repeats an
earlier claim's check, expectation, title and verdict. The verdict and
the count do not change; N claims sharing one check are one
verification counted N times, which a reader has to be able to see.
…the package

withKnowledgeMutation is reentrant per async context, but the
AsyncLocalStorage behind that was module-private. A consumer holding the
store lock through its own wrapper could neither observe nor enter the
scope, so calling any lock-taking function from this package inside its
wrapper self-blocked against a lock the consumer already held. The only
way out was to hold the lock by a private path and then forbid itself
from calling this package while holding it, which is how a second lock
over the same lockfile gets written.

isKnowledgeMutationHeld(root) reports whether this async context already
holds the root. runInKnowledgeMutationScope(root, hold, body) enters the
scope on a lock the caller took by its own path: inside it, every
lock-taking function in this package sees the root as held and runs
inline.

The owning path is now one function shared by a lock this package
acquires and a lock the caller supplies, so the externally held lock
gets the file transaction, the epoch, and the assertOwned checks at the
same points rather than a weaker subset. The hold is required rather
than assumed, because entering a scope for a lock the caller does not
hold is the one way to defeat the single-writer rule.

staleMs now documents what its window costs: the holder heartbeats at a
third of it, so it also bounds how long a crashed holder wedges every
other writer. Measured holds are 9 ms for one page, 586 ms for a
50-page batch, and about 390 ms for the largest store's longest
promotion, against a 15-minute default.
Two exported shapes moved: CheckExecution gains optional timedOut and
ClaimGrade gains optional duplicateOf. The surface record classifies a
shape change as breaking whatever it did, so the boundary is major.

A claim whose check contains its expected value is also refused where it
was recorded before, at both the record and the grade boundary.
…enforced

verifyGradeableEvidence bounds the check with timeoutMs, but a process
killed at that bound reports no exit status of its own, so runBash
returned 127 and the output was compared to the expectation like any
other failure. The deadline rule in gradeFor could therefore never fire
on this package's own execution path.

runBash now sets CheckExecution.timedOut when its budget did the
killing, and separates that from a caller's abort, which is a different
verdict.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 6d46826d

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

tangletools · auto-approval · reason: drewstone_author · 2026-09-01T17:16:10Z

@tangletools

Copy link
Copy Markdown
Contributor

⚠️ Review Interrupted — 6d46826d

The review runner stopped before publishing a final verdict: webhook_restarted.

State Detail
Interrupted webhook restarted

No review verdict was produced for this run. Trigger a fresh review on the current PR head if the PR is still open.

tangletools · #182 · model: kimi-for-coding · updated 2026-09-01T17:17:32Z

@drewstone
drewstone merged commit c5fcf54 into main Sep 1, 2026
2 checks passed
@drewstone
drewstone deleted the feat/grader-guards-and-detached-checks branch September 1, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(claim-evidence): three grader guards in gradeFor — unreachable input on a zero exit, deadline kill, precise expectation

2 participants