Conversation
…ists to catch The four mutants shipped with #3925 cover the detector, the verdict, an unlocatable reply and a reply cut at `You:`. None of them covers the state the commit was written to fix: the extractor bypassed and the whole transcript judged raw again. aprender-3e ran exactly that by hand while folding #3926 and confirmed it is caught. An unnamed mutant that happens to be caught is not a guard -- the next refactor to reintroduce raw judging would be noticed in review or not at all. Naming it means the self-test fails instead. self-test: mutant 5 (extractor bypassed, the transcript judged raw again) RED (expected) Refs #3925 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`run` was the last of the operator's four verbs whose generated text nothing read. Its capture was inspected for exactly two markers -- the fallback strings at :646 and `formatted_prompt=` at :657 -- and never for content. There was no `run_bad`. WHAT GOLDEN DOES AND DOES NOT COVER. aprender-3e challenged the first framing of this and was half right, so the corrected version is the one that matters: `golden_output` does have a real GPU leg, and it is what caught tinyllama. But it judges GGUF only (`output_verification.rs:754`, "the dense GPU leg judges GGUF only"), and a not-run leg still returns `Passed` (`golden_output.rs:667`). So on a `.apr` rung -- q4k, st, fp16 -- golden is a CPU-only verdict wearing a passing gate, and GPU content is covered by `chat` (#3897/#3925) and `serve` (#3886/#3921) alone. That gap is #3931; this commit closes the `run` half of it on every format. IT EXTRACTS RATHER THAN JUDGING RAW. `run --verbose` hands over kernel counts, VRAM figures and a list of hex pointers. Judged raw that is #3925 with different chrome -- though honesty requires saying the hazard is LATENT here: measured on the real gx10 capture, raw judging is clean, because the chatter is diverse and the repeated-fragment signal ignores diversity. The bound is taken because it is cheap and the chatter is not contractual, not because a false positive was observed. The reply is bounded by two lines we print, `Output:` and `Completed in <t>`. Absent or unterminated is its own RED naming the byte count, never a silent pass. The extractor is ONE function across all four verbs rather than a second parser. Guard: three real gx10 `run --verbose` captures in layer 3 (chatter -> clean, degenerate reply in the block -> bad, `Output:` never closed -> red), a second verdict fixture with `run` as the offender, and two more mutants. mutant 6 an unterminated run reply is taken as whole RED mutant 7 green stops reading the run verb RED Mutant 7 exists because mutant 2 removes the `output_bad` check for every verb at once and so cannot tell whether `run` was ever wired into `green` -- which is precisely how #3886 folded `serve` and left `chat` and `code` unread in the same structure. Stacked on the mutant-5 branch for numbering. 0.70.0 by aprender-3e's ruling: it can only turn currently-green rows red. Refs #3928, #3931 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ir meanings (#3932 re-review) The #3932 re-quorum (claude-sonnet-5, degraded: same-family) came back 2 PASS + 1 FAIL. Both findings were verified by hand and are fixed here (cop ruling B): 1. The explanation builder looped ("chat", "code"). #3928 added verb_ok(v, "run") to `green` and did not tell the builder, so a row red ONLY on run output printed `unknown`. That is the #3901/#3902 pattern this file's own comment warns about. The loop is now ("run", "chat", "code"). 2. The run extractor swapped exit codes 3 and 4 on the pre-existing chat/code paths, so a capture WITH an envelope was reported as missing one. 3 and 4 get their old meanings back, and an `Output:` that never closes gets its own code, 5, with a message saying what is missing. Guard (check_ladder_output_judged.sh): - check_run_reason: a row red only on run output must say "verb `run` RAN (rc=0) and produced bad output". It uses the same extraction as check_ladder_serve_verdict.sh. - The extraction table names WHICH piece is missing (red:envelope / red:reply / red:unclosed). A bare `red` could not see the swap. - Mutant 8: the builder drops `run`. RED. MUST-RED on the real pre-fix script (git show 25942b3:scripts/model_ladder.sh run through --script): 4 FAIL rows. reason:run-output says `unknown`; no-envelope comes back red:reply; envelope-but-no-reply comes back red:envelope; run-output-never-closed comes back red:envelope. The fixed script passes: the guard is green, all 8 mutants of --self-test go RED, check_ladder_serve_verdict passes (including --self-test), check_model_ladder --self-test has 0 bad cases out of 63, and check_no_pipe_into_grep_q is at its ceiling. Still open (PMAT-3928 item 3): the #3928 acceptance asks for a real gx10 CPU `run --verbose` capture. Only the GPU capture is in layer 3 so far. Refs #3928, #3932 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3928's "both backends" (PMAT-3928 item 3) Captured VERBATIM on gx10, 2026-09-23. Binary: apr 0.69.1 (7168e4b), sha256 01e821432fe8c46355040e1d1911e5a93251dd638873bcf01f191be03f2864de, snapshotted outside its target dir. Model: Qwen3-1.7B-Q4_K_M.gguf, sha256 b139949c5bd74937ad8ed8c8cf3d9ffb1e99c866c823204dc42c0d91fa181897. Command: the ladder's own invocation, `run --prompt "What is the capital of France? Answer briefly." --max-tokens 16 --verbose --no-gpu`, under gpu-q; rc=0. Its chrome is not the GPU capture's: a `verbose:` preamble, the formatted prompt and the raw decode. It extracts to "The capital of France is Paris." and classifies clean. Stated plainly, as a re-review lane found: judged RAW (mutant 5), BOTH real run captures still classify clean. They pin that real chrome is not flagged; they do not by themselves detect a bypassed extractor. Mutant 5 is caught by the chat and truncation rows, and the self-test still reports it RED. Refs #3928, #3932 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift
added
kind:code
Contributor
Author
Contributor
Author
|
fold source for the 0.70 integration branch; branch kept at Closed under the operator 1-day PR rule (2026-09-24); the rule does not wait on #4046. aprender-6c folds from the branch ref above into the 0.70 integration branch (built off main after #4046 merges). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. 0.70.0 by aprender-3e's ruling — do not fold into 0.69.1. It can only turn currently-green rows red, and
runis the leg the receipt'sgreenhas always depended on.Stacked on
fix/3925-name-the-raw-judging-mutant(#3929) for mutant numbering; rebase after that folds.Closes #3928. Related: #3931.
The gap
runwas the last of the four verbs whose generated text nothing read — inspected for the fallback markers andformatted_prompt=, never for content. Norun_bad.The framing, corrected in review
aprender-3e pushed back and was half right.
golden_outputdoes judge content per backend — it is what caught tinyllama. But its GPU leg judges GGUF only (output_verification.rs:754) and a not-run leg still returnsPassed(golden_output.rs:667), so on.aprit is a CPU-only verdict wearing a passing gate. That is #3931. This closes therunhalf, on every format.Honesty about the hazard
The ticket originally said judging
run --verboseraw would be "#3925 again". Measured, it is not — yet: raw judging of the real gx10 capture is clean, including the hex-pointer line (0 four-byte fragments repeated even twice). The chatter is diverse and the repeated-fragment signal ignores diversity. The bound is taken because it is cheap and the chatter is not contractual.Guard
Mutant 7 is separate from mutant 2 deliberately: mutant 2 removes the
output_badcheck for every verb at once, so it cannot tell whetherrunwas ever wired intogreen— which is exactly how #3886 foldedserveand leftchatandcodeunread in the same structure.bashrs lint: 0 errors.