Conversation
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
noahgift
added a commit
that referenced
this pull request
Sep 22, 2026
…ther than exempt check_apr_bin_pinned.sh failed guard-cargo on PR #3835 with two hits that came in with the CRUX slices fold. Both are FALSE POSITIVES: check_crux_inference_judge.sh:444 ok "... distinct GREEN cells; apr serve's backend unverified" crux_inference_dogfood.sh:709 "thinking ON (apr has no toggle until #3723)", Neither is a command. The first is an assertion MESSAGE, the second a descriptive row in a `not_covered` JSON list. In both, what the pattern read as a command opener -- `;` and `(` -- sits INSIDE a quoted string, which a line-oriented regex cannot see. Note that `"apr serve's backend is unverified..."` two lines above the second hit does NOT trigger, because `"` is not an opener and `(` is: the class is specifically an opener character appearing inside quoted prose. FIXED BY REWORDING THE PROSE, NOT BY ADDING CASE-TABLE EXEMPTIONS. An exemption row would have to teach the pattern to ignore `;`/`(` before `apr`, and those are exactly the openers a real violation uses (`cd foo; apr qa`, `(apr qa &)`). The guard is right to be strict here; the strings were just unlucky. Both rewordings are semantically identical and now word-preceded, which the case table already proves safe (`aprender-core builds apr eventually`). Verified the 709 string is descriptive and not compared: it is an element of the `not_covered` array in the receipt metadata dump, and `grep -rn "thinking ON"` finds no comparison against it. check_apr_bin_pinned.sh rc=0 check_apr_bin_pinned.sh --self-test rc=0 (78 regex cases, 9 surface probes) check_crux_inference_judge.sh 44 ok, 0 broke bash -n scripts/crux_inference_dogfood.sh ok Follow-up filed separately: the opener class does not account for an opener inside a quoted string, which is a latent false-positive source on any script that describes apr in prose. CLAUDE.md already records that this pattern class "has been wrong five times"; this is the sixth, and the first where the right fix was the call site rather than the regex. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
Review coverage of this batch, stated rather than implied
Why the batch cannot be quorum-reviewed as one diff2.89 MB. Every review lane available to us is bounded far below that: agy lanes took A code-only head ( What each row actually carries
This is the batching doctrine working as designed for implementation and thin for quorum: the author stops at the receipt and the cop folds, so the review unit is the row, not the batch. Ten rows carry implementation receipts with measurements, mutants and named deviations. Only one carries three independent PASSes. That is a real gap and I am naming it rather than papering over it. What is standing in for it on the gate-critical rowsNot an argument that quorum was unnecessary — an account of what was measured instead:
Known-red, by name
🤖 Generated with Claude Code |
6 tasks
Contributor
Author
Blocked on one operator decision: two guards exist and no workflow names themSeven reds have been cleared on this branch tonight, each one hidden behind the previous because Actions steps are fail-fast. The last one cannot be cleared from a commit.
Its rule is "every Both available remedies are outside what I may do autonomously:
The same ask covers #3904, whose two So the consolidated decision is: wire these two existing guards into CI, or park them out of the release. Either is fine by me; neither is mine to take. Everything else on this branch is green and measured
Complexity ratchet: PASS, One thing a tag-signer should knowSyncing the capability mirror changed the binary — Measured rather than argued: the two types that changed And a correction to my own reasoning, from aprender-45: widening a whitelist is not universally safe. A red→green flip on a deferred row turns the gate red, because the deferral that excused it goes stale and the amnesty check refuses it ( |
This was referenced Sep 23, 2026
… into release/0.69.1-batch-2
…pare test did not compile (#3956) 1. try_safetensors_cuda_backend ran reject_unsupported_ignore_eos BEFORE `state.safetensors_cuda_model()?`. It is dispatched ahead of the CPU quantized backend, so in any --features cuda build every `ignore_eos: true` chat request got 501 "not supported by the SafeTensors CUDA chat backend", even with no SafeTensors model loaded (GGUF CPU included). The refusal now runs after the residency check. try_apr_transformer_backend had the same ordering and gets the same fix. Must-RED: api::tests::usage_finish_3718 on base fc942f6 under cuda: 0 passed / 5 failed (left: 501, right: 200). After: 12/12 across usage_finish_3718 + ignore_eos filters, cuda and cpu. The PERF-039 fail-closed pair (registry fallback still 501s) stays green. 2. gpu_cpu_trace_compare.rs: add the missing `lm_head_tied: false` (E0063). It now compiles, runs, and FAILS on a real GPU/CPU divergence at layer-0 QKV (std 4.44 vs 0.30), 91.9% logits L2. Filed as #3975 and deliberately left RED, not #[ignore]d. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…3964) `/tmp/apr-gpu.lock` coordinates only the processes that take it. Ollama's llama-server is a system daemon that loads a model on any request and holds it for its keep_alive; it never takes the lock. On 2026-09-23 it arrived MID-RUN (1328 MiB) during a device A/B whose "card clear" check had passed at the start -- the same process and footprint that poisoned a cuda suite into 34 failures a week earlier. A clear card at the START proves nothing about the run. scripts/lib/gpu_exclusive_run.sh: 1. waits for an EMPTY card WITHOUT holding the lock -- holding the fleet lock while idle-waiting on a process that ignores it blocked two queued sessions and bought nothing; 2. takes the lock and RE-CHECKS the card; 3. samples every GPU process by FULL PATH every 0.1 s for the whole run (1 s missed a sub-second fault A/B; nvidia-smi answers in 10-20 ms); 4. exits 75, CONTENDED, if anything outside GPU_OWNED_PREFIX appeared -- a refusal, never a verdict. A run no sample caught exits 75 as UNVERIFIED: exclusivity nobody observed is not exclusivity. Two defects in its first release, both found on real runs: - It falsely refused two clean runs. The test binary reads "<pid>, [No data]" in nvidia-smi as it exits, and path-only classification called that a foreigner. A nameless sample is now ours only if that pid was already seen under our path; an unknown one stays foreign, conservatively. - Every signal handler EXITS. A `trap ... TERM` whose handler does not exit makes bash run it and carry on -- a stopped runner restored its file and went back into its wait loop, where it would have run a planted fault against the restored kernel and reported the fault as SURVIVING. Self-test: 8 rows, GPU-free (a stub nvidia-smi plays the card). Mutants killed, each by its own row: the mid-run foreign check, the never-clears refusal, the zero-sample rule, and the own-pid-at-teardown rule. Not covered by a dedicated row: the post-lock re-check, which is exercised only as a backstop (a mutant bypassing the pre-lock wait was still refused by it). Refs #3964, #3953 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…telisted (#3953) One of four types blocking Qwen3.5-0.8B-UD-IQ2_XXS (IQ2_XXS x95, IQ3_XXS x24, IQ2_S x5, Q2_K x3). This kernel alone does not turn that row green. The whitelist flip is deliberately absent: it lands in one combined admission PR with IQ3_XXS (#3963) and Q2_K (#3960) after #3969, so three rows do not rebase over the same lines three times. 82 bytes / 256: d, qs[32], signs[32], qh[8], scales[8]. One warp lane per (sub-block, group) as in IQ3_S; every field a lane needs is a single byte, so no unaligned load. The 10-bit index takes its high 2 bits from (qh[ib] >> 2l) & 3; the scale nibble is selected by l >> 1 (the reference's db[l/2]), NOT l & 1. THE GRID IS GENERATED, NOT COPIED. The 2048 u32 words are emitted from IQ2S_GRID at generation time, so the PTX table cannot disagree with the CPU oracle's -- IQ3_S's 512 hand-copied numbers went uncompared until a later test. KMASK_IQ2XS[j] == 1 << j is pinned, so reading a sign as bit j cannot silently desynchronise. ORACLE STRENGTH FIRST. The generated bytes are shown to use the qh high bits, two different scale nibbles per sub-block, and set sign bits -- before any fault relies on them. A fault on a mechanism the data never reaches would pass for the wrong reason. A/B at EVERY (k, n) the model uses for type 22 -- one shape, k=3584 n=1024, 14 super-blocks, which a 2-block synthetic test never reaches -- on synthetic and real bytes, per-row |gpu-cpu| / sum|w||x| <= 1e-5, output pre-filled with NaN: synthetic k=512 n=48 0.000e0 synthetic k=3584 n=1024 0.000e0 real bytes, all 5 tensors 1.1e-7 .. 1.9e-7 The real-bytes test proves the bytes ARE the tensor (every f16 d finite and small): with a wrong offset, GPU and CPU would read the same wrong bytes and agree perfectly. Four faults, each on a different mechanism, RED on real bytes: signs ignored (6.4e-2), qh dropped (6.0e-2), wrong nibble (3.4e-2), grid lo/hi swapped (6.1e-2). Values bit-identical across every run. Every run EXCLUSIVE under scripts/lib/gpu_exclusive_run.sh. Also no_generator_leaks_into_another_kernels_ptx_3953, at an honest scope: a quoted leak into a plain r"..." literal is a COMPILE ERROR (its first doc comment said otherwise and was wrong), so it covers what the compiler cannot -- a quote-free fragment, a PTX body duplicated inside one literal, and any future r#"..."# literal. Proved on plants that compile. fmt rc=0; clippy -p aprender-serve --lib --features cuda -D warnings: 0 errors. Full receipt: docs/audits/impl-3953-iq2s-gpu-gemv-receipt.md. Refs #3953, #3964, #3884 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…urce-weight drivers (hf, vllm) Cop (critical path, 2026-09-23): dd's #3962 certification on gx10 was dominated by the drivers reloading the model on every call (a vLLM engine start per prompt). Both drivers gain `gen-batch --batch <jsonl>`. Each line is {prompt_id, verb, messages, thinking, max_tokens}, and every line runs against ONE engine load: vLLM LLM(...) or `vllm serve`; hf one loaded model or one `transformers serve`. `gen` is now a batch of one through the same run_batch, so there is one code path. Row contract v1 is unchanged: exactly one row per item, in order, plus an additive `batch: {id, size}`. The #3971 content check runs once per load (verified_source runs inside the loader). Behaviour, each a case: - per-item max_tokens and thinking - a failed load refuses EVERY item with the load's own reason (vLLM: the engine core's root error) - one item failing does not take the others down - an invalid item is refused by name while the rest run - a batch holds ONE interface: mixing in-process (run, chat) with serve (serve run, code) would put a second engine on the card, so the other interface's items are refused by name - vLLM's preflight refusal fans out and loads nothing Oracle: scripts/lib/crux_batch_cases.py, 8 cases shared by both drivers, with counting fake loaders so "N items, ONE load" is itself asserted. vllm 24/24 (plus its cache/root-error cases), hf 8/8. Mutants, all killed: reload per item (vllm 3 broke, hf 3 broke); the mixed-interface refusal removed (1 broke). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…o release/0.69.1-batch-2
…3964) `/tmp/apr-gpu.lock` coordinates only the processes that take it. Ollama's llama-server is a system daemon that loads a model on any request and holds it for its keep_alive; it never takes the lock. On 2026-09-23 it arrived MID-RUN (1328 MiB) during a device A/B whose "card clear" check had passed at the start -- the same process and footprint that poisoned a cuda suite into 34 failures a week earlier. A clear card at the START proves nothing about the run. scripts/lib/gpu_exclusive_run.sh: 1. waits for an EMPTY card WITHOUT holding the lock -- holding the fleet lock while idle-waiting on a process that ignores it blocked two queued sessions and bought nothing; 2. takes the lock and RE-CHECKS the card; 3. samples every GPU process by FULL PATH every 0.1 s for the whole run (1 s missed a sub-second fault A/B; nvidia-smi answers in 10-20 ms); 4. exits 75, CONTENDED, if anything outside GPU_OWNED_PREFIX appeared -- a refusal, never a verdict. A run no sample caught exits 75 as UNVERIFIED: exclusivity nobody observed is not exclusivity. Two defects in its first release, both found on real runs: - It falsely refused two clean runs. The test binary reads "<pid>, [No data]" in nvidia-smi as it exits, and path-only classification called that a foreigner. A nameless sample is now ours only if that pid was already seen under our path; an unknown one stays foreign, conservatively. - Every signal handler EXITS. A `trap ... TERM` whose handler does not exit makes bash run it and carry on -- a stopped runner restored its file and went back into its wait loop, where it would have run a planted fault against the restored kernel and reported the fault as SURVIVING. Self-test: 8 rows, GPU-free (a stub nvidia-smi plays the card). Mutants killed, each by its own row: the mid-run foreign check, the never-clears refusal, the zero-sample rule, and the own-pid-at-teardown rule. Not covered by a dedicated row: the post-lock re-check, which is exercised only as a backstop (a mutant bypassing the pre-lock wait was still refused by it). Refs #3964, #3953 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 0d000a7)
…eeping BOTH intents, A3B deferral retired Merges PMAT-3714-r2-on-v2@6f6fa9a83 into release/0.69.1-batch-2 on the operator ruling of 2026-09-23, relayed verbatim by aprender-6a (cop): "fold in MoE". TEXTUAL CONFLICTS (4) - infer/inference_result.rs: release threaded `gpu_attempted` (#3826); MoE added the CUDA dispatch returning (tokens, used_gpu). Kept both: the MoE dispatch attempts CUDA exactly when this is a cuda build and --no-gpu is absent, so gpu_attempted is derived from those two facts. That is what makes #3817 VISIBLE: a GPU that was tried and refused reads fell_back:true instead of as a quiet CPU answer. `canonical_arch` is kept (used for context clamping); `moe_forward_handles` is taken as the single dispatch predicate (identical to canonical_arch == "qwen3_moe", so no routing changes). - qa.rs: both sides independently fixed the same #3817 zero-byte-JSON abort. Release's comment and message are kept, with MoE's measured duration instead of ZERO. - golden_output.rs: release's cuda_device_present() helper (identical to MoE's inline check) plus MoE's #3750 8-byte magic read, so the whole model is no longer read into memory. Second hunk: MoE's mapped header plus release's #3870 golden_prompt_tokens(). The MoE side had REINTRODUCED `vec![SpecialTokens::qwen2().bos_id, 9707]`, the exact hardcode #3870 removed. - docs/roadmaps/roadmap.yaml: resolved with the repo's own driver, scripts/lib/roadmap_merge.py. Verified as a true union BY ID: release 1044 + MoE 2 = 1046, no duplicates, nothing missing from either side. SEMANTIC CONFLICTS (merged cleanly, would not have worked) - golden_output.rs:816 `gguf_model.as_ref()` did not compile: #3750 made gguf_model a borrow of the map. Passed as-is (Option<&T> is Copy). - capability::no_cuda_forward_reason still REFUSED qwen3_moe on CUDA ("lands in 0.70.0"). The MoE branch never touched the file, so the merge kept the refusal silently, and it gates `apr run --gpu` (run_entry.rs) and qa's capability_match. The MoE branch's "run/qa green" receipts were therefore taken on a tree WITHOUT this refusal. The function's own doc prescribed the fix: "deleting the arm here is what turns the refusal off". NARROWED, NOT DELETED: normalize_architecture folds the Qwen3.5-MoE spellings (qwen3_5_moe, Qwen3_5MoeForCausalLM, Qwen3_5MoeForConditionalGeneration) into "qwen3_moe", and Qwen3.5 MoE has Gated-DeltaNet/SSM layers the qwen3moe forward does not run. Those spellings were protected only incidentally by the blanket refusal; deleting it would have routed them to a forward that cannot run them. The refusal now matches the raw Qwen3.5-MoE spelling (qwen35moe, the GGUF string of Qwen3.5-35B-A3B, included). Tests INVERTED, not deleted: every_qwen3moe_spelling_has_a_cuda_forward_and_is_routed_to_it (refusal + dispatch agree) every_qwen35_moe_spelling_is_still_refused_by_name gpu_forced_on_qwen3moe_reaches_its_cuda_forward gpu_forced_on_qwen35_moe_is_still_refused_by_name removing_the_qwen35_moe_refusal_would_route_a_hybrid_to_the_wrong_forward (owned mutant) Mutant planted (narrowing disabled): 3 tests go RED across both crates; restored byte-identical. docs/GPU-SUPPORT.md regenerated by its own generator. The conflated "Qwen3 / Qwen3.5 MoE" row is split: qwen3_moe -> yes, qwen3_5_moe -> refused, with the reason. CONTRACT - model-capability-ladder-v1.yaml: inventory.deferred is now {}. `pv validate`: 0 errors. NOTE, and it is NOT decided here: `*A3B*` also covered Qwen3.5-35B-A3B-UD-IQ4_XS.gguf (arch qwen35moe) on lambda. Nothing in this fold gives it a CUDA path, so it is now a required RED row. That is a scope call for the cop/operator. VERIFIED ON THE MERGED TREE (private target dir) - #3714's own device tests, once per file, 0 SKIP lines: Qwen3-Coder-30B-A3B & Qwen3-30B-A3B-Instruct-2507, lambda RTX 4090: e2e cosine 1.000000 and argmax equal at every position to 64; injected routing faults DropTopExpert / UniformWeights / WrongExpert -> REJECT by both the exact-reference check and runtime F2 (min cosine -0.44..0.53, 14..65/65 argmax mismatches); clean -> ACCEPT 0/65. (Ran on the fold binary before the capability narrowing; that change touches neither the forward nor the parity code.) - Full lib suites, no cuda (CI's profile): apr-cli 7417/0; aprender-serve 16015 passed. 2 `should panic` tests fail in --release only; they fail identically on the untouched base in release and pass on this tree in debug (debug_assert). - Ladder guards --self-test: check_model_ladder, output_judged, provenance, serve_verdict, serve_backend_record, all 0. Judge verdict delta vs base: only the MoE rows moved from DEFERRED to owed. - fmt clean; clippy --lib -D warnings clean for both crates. The 60 errors under --features cuda are all in aprender-train (entrenar), none in this fold's files. Refs #3714, #3817, #3826, #3870, #3750 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… against it and the AC, not a typed count (#3965) Cop ruling on 2b75f93: fix ship-006 on this branch, and DERIVE the gate count from apr qa's own registry. THE REGISTRY. There was none: the gate set existed only as the sequence of dispatch_gate calls in run_qa. New QA_GATES (12 names). run_qa checks what it EMITTED against it on every run; a mismatch (a gate dropped, added, or emitted twice) is a FAILED `gate_registry` gate, not a warning. The report publishes it as `gates_registered` (serde default, so old cached reports still load, and consumers treat empty as "cannot check"). No new CLI surface: `apr qa` takes a required FILE, so a list flag would have forced an invasive clap change, and a report that carries its own registry lets a consumer check emitted-vs-registered from ONE binary. Measured on real runs of all three formats (tinyllama GGUF, the BF16 .apr, 0.5B SafeTensors): 12 gates emitted = 12 registered, no false gate_registry failure, and no skipped gate claiming passed. Unit test gate_registry_mismatch_catches_drop_add_and_repeat. SHIP-006. REQUIRED_GATE_COUNT=8 compared against 12 emitted gates, so the discharge could never pass. Its historical "pass" counted skips: the contract's 2026-05-10 note reads "All 12 gates pass (6 executed, 6 skipped)". A naive fix of "all 12 must pass" is equally wrong in the other direction: classifier_head is skipped unless requested, so that could never go green either. Per the cop's lattice ruling (a skip is RED only when the claimed capability REQUIRES the gate): REQUIREMENT comes from the AC: the 8 gates AC-SHIP1-006 names (qa.md §3), each of which must be REGISTERED, EMITTED once, EXECUTED and PASSED. PRESENCE comes from the binary: every other registered gate must be emitted and not FAILED; a skip there is neutral. A report without gates_registered is FAIL (cannot derive), never PASS. The count is derived (${#REQUIRED_GATES[@]}), never typed. Every reason it is not a PASS is named. Also: it captured `apr qa … 2>&1`, merging diagnostics into the JSON, so one stderr line broke the parse. stderr now goes to its own file. GUARD scripts/check_ship006_gate_registry.sh runs the SHIPPED script against a fake apr, one crafted report per case, each FAIL checked for its OWN reason: full -> PASS · stderr-noise -> PASS · dropped-required -> FAIL (emitted 0 times) dropped-other -> FAIL · skipped-required -> FAIL ("a skip is not a pass") skipped-old-encoding (passed:true,skipped:true) -> FAIL · failed-other -> FAIL no-registry -> FAIL --self-test runs the PRE-FIX script (a foreign oracle, not a mutant of mine): it FAILS the healthy 12-gate report, exactly the stale-count defect. Wired (check_guards_are_wired rc=0). bashrs: 0 errors. The jq judge lives in a quoted heredoc: inside single quotes, bashrs read jq's `$req[]` as six bash array expansions. Verified: apr-cli --lib 7405 passed 0 failed; fmt 0; clippy -D warnings 0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…release/0.69.1-batch-2
…ubstrings (#3839) libtest's --skip is a substring match on the FULL test path. `make coverage` passed 19 bare substrings, which removed 2,713 lib tests from the instrumented run (all=86,807, kept=84,094, measured with --list at fc942f6). 1,352 of them matched only on a MODULE path, never the test's name. By pattern: gpu_ 1,147 (includes every wgpu_*), falsification 1,084 (whole modules), chaos 135, load_test 112, cuda 106, heavy 67, slow 48, disconnect 29. Their code then read as uncovered, so the gate measured a subset over the whole denominator, and #2307 / pmat --coverage-gaps ranked well-tested modules as the top gaps. Measured which of them need skipping: all 2,700 unique names run under nextest on intel (no working NVIDIA driver), 30 s slow period, 120 s terminate. 2,702 passed, the slowest in 41 s, 121 s wall. The ONE timeout is apr-cli test_llm_band::cuda_without_the_server_feature_is_refused. It is not a GPU test: build_provenance hashes current_exe twice per call, and the test calls it twice, i.e. four passes over a 494 MB debug binary. The skip list is now scripts/coverage-skips.txt, one exact path per line, each with its measured reason, passed as `--exact --skip <path>`. Checked on the real apr-cli binary: all 7,416 → exact entry 7,415; `--exact --skip gpu_` 7,416 (cannot over-match); old substring gpu_ 7,343. COV_FLOOR is unchanged. Coverage before/after numbers follow in the next commit, measured with the same instrumented build. Refs #3839 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…, ALL_WRONG is RED (#3957 F6) Implements the quorum-revised CRUX oracle (#3957 comment 5790953724, with the cop's per-format correction). A cell is GREEN or RED; GREEN only when: 1. apr is correct under scripts/lib/crux_oracles.py (#3962, aprender-dd): the constrained <answer>X</answer>, think blocks stripped, an UNCLOSED think block RED (Q5). A v1 expect_any prompt is a substring oracle: RED (Q3). 2. SAME-REPRESENTATION (Q2): the engines on the identical weights (GGUF: ggml family = llama.cpp+ollama+llamafile, ONE vote; SafeTensors: hf/vLLM) answered, agree, and equal apr's EXTRACTED answer. A split is RED (Q1). A .apr has no such engine (proven only through the ladder's F8 chain). 3. GROUND TRUTH: a bf16 control (hf/vLLM) answered, every one correctly. degenerate() now also catches a multi-character token loop (#3971). Controls: one POSITIVE per (model, host, verb, thinking); a NEGATIVE per verb (the judge plants the prompt's into a GREEN control cell and must see RED). J2: a v2 set is judged only under its certification receipt (--certification). Pre-fix evidence, judge @ 5c59389 (functions called directly): PRE-FIX F6 evidence, judge @ HEAD 5c59389 GREEN apr_correct=True comparator answered WRONG, apr right -> must not be GREEN ALL_WRONG apr_correct=False everyone wrong -> must be RED GREEN apr_correct=True answer only inside <think> -> apr must not be correct GREEN apr_correct=True negation 'not Paris' -> must not be correct GREEN apr_correct=True 'Parisian' (no word boundary) -> must not be correct degenerate('NavController'x12) = False (must be True: a token loop is not an answer) Live: aprender-83's #3971 receipt (vLLM "NavController"x12, cause a poisoned HF cache) was PASS 4/4 GREEN; re-judged by this judge: RED 4/4, "ground-truth control FAILED (vllm: degenerate ...)". Case table ported to a v2 fixture prompt set + fixture certification (the golden_output.rs drift check still reads the real v1 set): 107 ok, 0 broke, incl. 17 new F6/J2 rows and 8 judge mutants, each required to break its NAMED row. Two first drafts were killed for the wrong reason and are fixed: the mutant copy could not find crux_prompt_certify beside __file__ (every row broke), and the negative-control case also declined on a stale certification. Merges aprender-dd's feat/3962-crux-prompt-certification (oracle lib, v2 set, certifier) and aprender-83's PMAT-3952-crux-vllm-judge (vLLM comparator), whose hunks this builds on. Refs #3957 #3962 #3971 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…wers and was cut off before the tag (#3962) Measured on gx10: 7 of 21 thinking-OFF cells on Qwen3.5-2B-BF16 were no_answer_tag with the correct value in a truncated worked solution. The runner records max_tokens per row and re-measures a changed budget. Refs #3962 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…elease/0.69.1-batch-2
…uant sha (#3962) - check_crux_oracles.sh now refuses drift. Every golden_questions() case must open an `answer` prompt whose expect is one of its patterns, or be named in golden_excluded with a reason. A stale exclusion, and a parse that finds zero cases, are refusals too; all four were planted and went RED. - golden-paris adds the exact golden question under a new id. Existing prompts are unchanged, so rows already measured for them stay valid. - The certification receipt carries admitted_by_sha {quant sha256: [ids]}, so the judge can check admission per cell (aprender-6c [8b6b78]). Refs #3962 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…he unpinned rule rides F6's reasons aprender-6c's F6 branch already merged this branch at a545a77. This merge brings in F6 on top of the three commits made since: unpinned→RED, and the #3964 lock check with chat coverage. Conflicts, resolved to ONE mechanism: - judge: F6's judge_cell returns (verdict, ok, reasons, extracted). The unpinned rule now sets RED and APPENDS "oracle unpinned: …" to F6's `reasons`. My parallel `verdict_reason` field is dropped, since two reason fields would disagree sooner or later. - case table: the unpinned cases use F6's fixtures (<answer> tags, a llama.cpp same-representation row). The apr-right case is now a CONTROLLED PAIR: the identical cell with vllm pinned is GREEN, and unpinning vllm alone turns it RED with the unpinned reason. Under F6 the old single case would have been RED anyway, for "no same-representation engine", so it no longer measured the rule. check_crux_inference_judge: 112 ok / 0 broke. Mutant (override off): 2 broke — the unpinned cells fall back to F6's "no ground-truth control", so the reason assertion is what catches it. check_crux_ollama_in_lock: 3/3 on the merged dogfood. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ust-RED case table (#3962) SERVE: the route universe is the live server's own `GET /` index (router.rs advertised_routes), never a grep plus a hand-written filter. Every generation route is driven x every mode its wire has x every serve prompt: - verb keys "serve run" / "serve stream", plus an additive `route` field; - raw-prompt routes are rendered by llama.cpp /apply-template, and the sha256 is recorded; - stream terminal events are required ([DONE] / event: done / done:true), so a truncated stream, zero deltas, an empty text or a missing finish_reason is RED. An unclassified route and a router with no index are RED rows, never absences. CODE: `apr code -p --output-format json` in an empty tmp project, reply copied raw. apr code has no backend, max-tokens or thinking control (it hardcodes `apr serve --gpu`), so the rows say so and the cpu lane is refused. The comparators are llama.cpp and ollama over the OpenAI chat route, plus the plugin `gen --verb code`. scripts/check_crux_serve_code.sh: 33 rows plus 8 mutants, each mutant deleting the check a must-RED row relies on, and each killed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… release/0.69.1-batch-2
…erdict (#4004, #3957 F9) aprender-83: #4004's CPU reference comes from greedy-only shards (thinking OFF, ctl-2plus2), because the certified 4096-token ON cells are impractical on CPU. Measured: collect then writes 0 cells and verdict DECLINE ("no cell was measured") with greedy[] populated. Before this, the F9 judge IGNORED its greedy (DECLINE is skipped), and the cell join FAILED the cut on it (it reads every *.json). A receipt marked `greedy_only: true` with no cells is now skipped by the cell join (a note, no failure) and read by the F9 judge despite its DECLINE, still bound to the cut sha. A greedy_only receipt that carries cells is refused: it is a lane verdict or greedy evidence, never both. +1 green, +1 red case, 3 mutants each killed by its case. self-test: 135 case(s), 0 bad. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… hybrid leaves the chain unchanged (#3962) every_raw_route_answers_from_the_hybrid (real Qwen3.5-0.8B-Q4_K_M, CPU): /generate, /batch/generate, /realize/batch, /stream/generate, /realize/generate each 200, no echo, and clean_chat_output(answer) == the one-shot dispatch on the same rendered prompt. Mutant (the arm declines): RED with the original '503 Model registry error: No model available' on /generate. a_state_without_a_hybrid_leaves_the_raw_chain_unchanged: all three entries return Ok(None) on a demo state. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…cells, and is named <host>-<backend>-greedy.json aprender-36 read in the code that model_ladder_crux.load_crux globs EVERY *.json in the crux dir and FAILS the cut on a DECLINE, and that the F9 judge skipped DECLINE receipts. So an unmarked greedy-only receipt would have failed the release. The contract, per 36 (judge side at fix/3957-f9-f10@09ea08424): top-level "greedy_only": true and "cells": []. The ladder's cell join then skips it, and the F9 judge reads its greedy[]. The orchestrator writes it to <out>/<host>-<backend>-greedy.json (never mistaken for the certified receipt) and marks it. It REFUSES to mark a merge that produced cells, or one whose greedy[] is empty (it would prove nothing). The judge's DECLINE is the expected verdict there, so the script exits 0 after a successful mark. Exercised on real receipts: a greedy-only one (8 greedy entries) is marked; one with 15 cells is refused (rc 1). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ase/0.69.1-batch-2
…to release/0.69.1-batch-2
…elease/0.69.1-batch-2
…haracters into U+FFFD (#3962) The raw SSE handler decoded each generated id alone, so a character spanning two byte tokens streamed as two U+FFFD - the defect a299890 fixed on the chat SSE path. raw_stream_token_events reuses LiveUtf8Deltas (now pub(crate)): a held id emits no event, the completing event carries the joined text and the last id. Tests: a_multi_byte_character_streams_whole, an_unfinished_character_at_the_end_is_ flushed_not_dropped; stream-related filter 14 passed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ntation (0.69.1 CRUX sweep RED)
The freeze sweep's chat ctl-code-add cell was RED in both thinking modes. apr chat's transcript
had no line with more than one leading space: Python indentation arrived as ONE space, while
llama.cpp kept four. Cause: clean_chat_response (crates/apr-cli/src/commands/chat.rs) ran
`while cleaned.contains(" ") { replace(" ", " ") }` over the reply, then trim()med it, which also
ate the first line's indentation. It was added in 8dd6e3b (PAR-201); byte-level BPE `ĠĠ` IS two
spaces, the very tokens indentation is made of.
Now only the blank lines around the reply and its trailing whitespace are dropped. Must-RED:
" return a + b" round-trips byte-identical, as does a multi-level indented function (with
<|im_end|> and surrounding blank lines) and a tab. It was RED on the unfixed code (left "return a + b").
Five existing tests ASSERTED the collapse (many_spaces, normalizes_spaces, multiple_bpe_spaces,
complex_combined, trims_whitespace); they now assert preservation, each with a comment naming the
defect. apr-cli chat tests: 290 passed.
Not changed, flagged: normalize_repeated_punctuation still caps runs of ! ? . at three in model output.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…a-cli output (#3962) aprender-83, freeze sweep (lambda, Qwen3.5-4B, ctl-code-add): llama answered the code cell right (rc 0, drive JSON) but engine_entry read verb 'code' comparator rows with the llama-cli echo parser, so it was 'not answered' and the cell went RED 'no ggml-family engine answered'. Comparator code rows now parse as serve JSON. Replay of the freeze manifest: code/off RED -> GREEN. code/on stays RED on apr exit 3, which is apr code's own refusal of --thinking on (#3723), not the judge. Case table 153 ok / 0 broke; mutant code-comparator-cli killed by 'J/code comparator json'. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ate_kwargs.enable_thinking / think) The model's OWN template is rendered with the request's enable_thinking (absent = OFF, #3801's default) through official_or_legacy, the one rule apr run and apr chat use: an ON the template cannot express is refused 400 by name; two spellings that disagree are refused at the handler entry; an unknown chat_template_kwargs key is refused at deserialization. OpenAI wire (chat_template_kwargs, think) and Ollama /api/chat (think). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…h carries one prompt (#3962) aprender-83's freeze sweep (lambda, Qwen3.5-4B and -2B, OFF, both modes): ctl-recall-7 on POST /api/generate was RED 'apr differs from the ggml family'. The drive sent the ollama_generate wire messages[-1] only, so apr was asked 'What number did I ask you to remember?' with no turn 1 and correctly said it had no access to past conversations, while its B4 oracle (llama /v1/chat/completions) had the whole history. Two questions, one cell. apr /v1/chat/completions and /api/chat answered <answer>7</answer> both turns. SINGLE_PROMPT_KINDS + history_refusal: drive REFUSES the pair (rc 4, never sent truncated); sweep skips it like a mode the wire lacks and records it in the plan's not_applicable. Rows: S7c must-RED, S7d one-turn still asked, W5 plan records the skip; mutant M16 killed. check_crux_serve_code 66 rows / 0 failed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…into release/0.69.1-batch-2
…d of refusing ServeLaunchOptions.think rides every request as chat_template_kwargs.enable_thinking. The up-front refusal is retired; the one remaining refusal is the embedded fallback (no apr serve), which has no thinking-ON path and says so rather than serving OFF as ON. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… (0.69.1 sweep, cop ruling)
normalize_repeated_punctuation capped runs of ! ? . at three in every chat reply ("Wait...." ->
"Wait..."). That is the same class as the whitespace collapse: chat rewrote the model's output
and run did not, so the two disagreed, and CRUX compared chat against a reference that rewrites
nothing. The function and its call are removed; only special tokens are stripped now.
Must-RED: "Wait...." and "!!!!" round-trip unchanged; RED on the previous commit (left "Wait...").
Five tests that ASSERTED the cap now assert verbatim, each with a comment. apr-cli chat tests: 290 passed;
clippy -D warnings and fmt clean.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…release/0.69.1-batch-2
…ceipt), never a receipt with holes gx10, 0.69.1 final sweep at c619ddd: the root fs filled mid-run (332 MB free of 916 GB). Every artifact of qwen35-27b-q4km failed to write, its row append to $ROWS failed ("printf: write error: No space left on device", model_ladder.sh:1145), an inventory record's append to $INV_ROWS failed (:1259), and the run CONTINUED. The only trace was stderr. The receipt it was heading for would have looked complete while missing a rung and a model the host holds, and the judge reads that receipt. The cell itself had been recorded as a MODEL FAIL ("the receipt row could not be built"). FIX, three layers, each a decline (exit 2: the ladder's existing "nothing trustworthy" code) that writes NO receipt: 1. ladder_append: every append to $ROWS / $INV_ROWS (all five sites) is checked, and counted. 2. ladder_disk_probe: a 64 KiB write that must read back, plus a free-space floor (LADDER_MIN_FREE_MB, default 1024). Run on $WORK and the receipt dir BEFORE any model is measured; at the start of every cell; and again right before a cell's row is kept. If the disk failed DURING a cell its artifacts may be truncated, so the row is not built from them. 3. The receipt goes to a temp file, python's rc is checked, it is parsed back, its rung and inventory counts must EQUAL the appends this run made, then it is renamed into place. It was written directly to its final path, so a failed write could leave a truncated gx10.json. Also: `WORK=$(mktemp -d)` was unchecked (a failed mktemp made the paths /rows.jsonl). GUARD scripts/check_ladder_write_errors.sh: the SHIPPED helpers against REAL failing writes (a file behind /dev/full is a true ENOSPC; a read-only dir a true EACCES), and the SHIPPED model_ladder.sh end to end with a fake apr: append-enospc -> exit 2 naming the file · append-readonly -> exit 2 · append-ok -> record present, counted · probe-readonly -> fails · probe-floor -> fails, names it · e2e-readonly-out / e2e-readonly-tmp / e2e-full-disk -> rc 2, NO receipt, `decline:` --self-test plants the old behaviour (an append that ignores its failure; a probe that always passes) and requires each to turn it RED: rc 0. FOREIGN ORACLE: the PRE-FIX ladder (c619ddd) given the same read-only --out did NOT decline. It began measuring and recording [FAIL] verdicts ABOUT THE MODELS (qwen2-1.5b-q4km, qwen3-1.7b-q4km, …) for an environment problem, until a 400 s timeout. The fix refuses before touching a model. Verified: every ladder guard and its self-test rc=0 (only_selection, output_judged, provenance, serve_teardown, serve_verdict, write_errors, serve_probe_evidence); check_guards_are_wired rc=0; bashrs 0 errors on both files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ard's models — one model's meta made the others "format-unknown" RED Measured on the freeze sweep (lambda, c619ddd): the host receipt read 47/140 GREEN, but every certified shard's own receipt read 30/34, 17/19, and so on. The merge took the FIRST shard's meta as-is, so the judge knew one model's format (4B-Q4_K_M), and every 2B and 4B-UD cell went RED "no engine but apr reads a format-unknown file". - The merged meta is now the first shard's, with `models` = the union over every certified shard's meta.models (deduplicated by sha256; greedy shards excluded), plus merged_shards. It is written as <out>/<host>-<backend>.meta.json. - --merge-only re-merges an existing run from <out>/shards.tsv without running anything. Re-merged the freeze sweep's lambda shards with this (no GPU): 124/140 GREEN. Every model/mode now matches its own shard (2B off 30/4; 4B off 30/4, on 17/2; 4B-UD off 30/4, on 17/2). The 16 REDs are the known, routed findings: apr chat whitespace (5), the code-comparator parse (3, 70's post-freeze fix), ctl-recall-7 on serve (6), and code ON refused by name (2). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…o release/0.69.1-batch-2
…into release/0.69.1-batch-2
…release/0.69.1-batch-2
Contributor
Author
Contributor
Author
|
fold source for the 0.70 integration branch; branch kept at Closed under the operator 1-day PR rule (2026-09-24); the rule does not wait on #4046. aprender-6c folds from the branch ref above into the 0.70 integration branch (built off main after #4046 merges). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
0.69.1 — the stoppers train for 0.69.0, as one integration batch
88 commits, one CI run, one queue slot (#3669's pattern, second use). Rows folded:
#3760, #3786, #3775, #3754, #3743, #3757, #3794, #3817, #3791 (+#3571 u1, #3595,
#3571 u2), #3801, #3801b, #3724, #3715 (Alfredo), #3782, #3828, CRUX slices 1+3+4
(#3739, #3795, #3797), plus d8's census/README regen.
The theme is gates that could not fail
0.69.0 shipped a model emitting empty output on CUDA as a note rather than a
failure, because one contract key said the rung was optional. This train flips that
key and then fixes what flipping it exposed.
Four release gates here are one defect wearing four faces.
contracts,pv lint,check_model_ladderand the clean-room contract test all resolve to a single missinggreen witness:
ladder-greenfiringmissingGreenHost: lambdaonqwen3-8b-q4km.resolve()unions green witnesses across versions, so one green 0.69.1 lambda receiptcloses all four and no committed receipt is edited. Measured, not predicted: a
synthetic green receipt was dropped in,
cargo test -p aprender-contracts --lib lint_passes_on_real_contractspassed (12 gates, rc=0), and the tree was verified cleanafter removal.
The entries worth reviewing are the gates that were green while being wrong:
"!"by substring match"!"is token id 0 in the Qwen vocab. Pre-fix,"!"-repeated passed at lengths 1..=11, caught only at 12+ (bytes.len() >= 12). Blind exactly where a dead-logit model failsmake contracts"$PV" lint contracts/ | tail -5tail's exit status — verdict printed, never enforced. Third time this idiom has shipped (#2336, #2360)run{run, chat, serve, code}(#3828)FALSIFY-README-003Sequencing that is load-bearing
model_ladder.sh:106-115takesVERSIONfromcargo metadataand line 441 stamps itinside the receipt, so a pre-bump run would have written
evidence/dogfood/models/0.69.0/<host>.jsonclaiming a version that already shipped.golden_output.passed.qwen3-8b-q4kmis being re-measured precisely because its0.68.2 lambda receipt reads
golden_output: "Empty output"— so under the pre-apr qa golden_output: the greeting case's "!" pattern scores degenerate output ("!!!!", token id 0) as correct #3782patterns, a fix that turned empty output into
"!!!!"would have scored CORRECT andwritten a GREEN receipt closing four gates on garbage.
commitandversionto HEAD bystring equality and R4 requires HEAD to be an ancestor of
origin/main.Known limitation, stated by name
qwen3moeon CUDA refuses before loading, names the architecture, cites #3714 and exits12 — instead of loading 18 GB, generating on the CPU and exiting 14 after the fact. The
MoE GPU forward is #3714, in 0.70.0. (#3817)
Verification
The four gates above are expected red until the ladder receipt lands in this PR.
🤖 Generated with Claude Code
keep-open: every
#Nabove is cited as PROVENANCE for a gate this batch fixes, not as something the batch closes. Fixing the gate that DETECTS a defect does not fix the defect — so #3714 (MoE CUDA, scheduled 0.70.0), #3715 (the cells producer, unbuilt), #3828, #3817 and the rest stay open on their own evidence. Per-issue dispositions are deliberately NOT asserted here: I have not verified, issue by issue, which of the twenty this batch's work resolves, and a wrongCloses #Nwould auto-close a live defect on merge. Anyone who establishes that a specific one is fixed should close it on that evidence, not on this PR's.ont-delta: none this batch adds no ontology type, shape or reason. Measured:
contracts/shapes.ttlcarries 29sh:NodeShapeonorigin/mainand 29 on this head, andont4b_shapes_gate.rsassertsshapes_n == 18unchanged — so no shape was added, and a shape added unnoticed is what that assertion exists to prevent. What DID change is instance data: 4 new contract files andcontracts.nt15863 -> 16117 triples (+254), re-extracted with the in-treepv(sha2564188c51a..., byte-identical to CI's own extraction). The batch's substance is making existing gates able to fail rather than extending the ontology, sononeis the accurate kind here and not an evasion of one of the other four.