feat(parity): L0-1b — apr parity --per-op names the first diverging op (1.5B: post_ffn_residual layer 26); the CPU Q8_K reference is the inaccurate side

[noahgift]

Row L0-1b (#2971, PP-066 #2873) — steps 0, 1 and 2. Not armed yet: receipt is status: partial until the gx10 twin is measured (fleet-verify).

Ticket

PMAT-1070 (minted in #3025). Issue #2971 (P0, inst:A).

Claim

apr parity <gguf> --per-op names the first op where the GPU-resident forward stops computing the function the CPU reference computes — per (stage, layer), over every prompt position (78 ≥ 64, I8), exit 0 with the table, the admission gate bypassed internally and recorded (never SKIP_PARITY_GATE). On the 1.5B it names post_ffn_residual layer 26 (min cosine 0.660150 @pos 0) and its lm_head row is the gate's own 0.9508; on the 7B it names nothing.

RED test

Commit 71a25c2: parity_per_op_table unit tests (forward order; downstream red is never the answer) and per_op_tap tests (selected stage only; unarmed writes and computes nothing; bypass thread-local). Live twin (POP-F-003): .pr/L0-1b/accept.sh legs A5/A6 on lambda.

Acceptance

APR=<cuda build> bash .pr/L0-1b/accept.sh → 6/6: table tests · tap tests · pv validate · arms doc · 1.5B names layer 26 with lm_head within 1e-3 of 0.9508 · 7B clean.

Step 1 arms (measured, docs/audits/l0-1b-arms.md)

graph off · FP8 decode off · FP8 all off · flash decode off · fused gate-up off · all off → 0.9508 @pos 0 in every arm; 7B 0.9986. No switchable GPU path is the mechanism.

The finding (measured; the receipt and the arms doc carry the tables)

Layers 0–25 agree at every op. At position 0 the residual carries a massive activation (dim 408 = −3664); the layer-26 FFN cancels it on CPU (ffn_out +3675.7 → residual 11.3) but not on GPU (4084.2 → 407). ffn_norm@26 is identical on both sides; ffn_swigl@26 neurons 2908/7035 differ by +14 %/+7.5 %. A float64 recomputation from the dequantised weights (.pr/L0-1b/step0/probe26.py) gives −1142.0/618.4: the GPU is within 0.5 % of the truth; the CPU (−996.4/573.6) is 13 % off and is reproduced to three decimals by Q8-quantising the activation per 256 elements. The CPU Q8_K reference is the inaccurate side on massive-activation tokens.

Step 2 — the fix (commit 361d685)

quantize::has_crushed_block (per 256-block max/second ≥ 8; basis: the measured criterion table — never on 77 ordinary positions × 28 layers, ≥ 20 on the first token's crushed blocks) sends that one matmul to the f32-activation kernel (quantize::direct_f32) through matvec_into_honest on the reference forward (QKV, o_proj, gate/up, down) and the fused gate+up driver.

	before	after
1.5B apr parity min cosine	0.950827 (1 < 0.98, 2 argmax≠)	0.999761 (0, 0)
1.5B --per-op first diverging op / lm_head	post_ffn_residual L26 / 0.950827	none / 0.999761
7B apr parity min cosine	0.998607	0.999580
GPU dump tree, GPU m=1 greedy streams on 5 local models	—	byte-identical
CPU decode 1.5B 64 tok n=3 (tok/s)	11.6 · 10.8 · 10.8	12.0 · 13.2 · 12.6 (no slowdown resolved; basis [U])

Mutation

• POP-F-003 revert twin (measured, same tree): has_crushed_block → false → the 1.5B table names post_ffn_residual layer 26 again, lm_head 0.950827, accept.sh A5 RED; the mutant's CPU dump tree is bitwise identical to the pre-fix tree (24,258 files) — the routing refactor changes nothing by itself. Restored → 6/6.
• Unit-level (in the contract): reverse the forward scan → ordering test RED; ignore should_save → tap test RED; has_crushed_block 4 tests (the layer-26 block, an ordinary block, one-hot, all-zero).
• CI RED→GREEN pair: no guard step is added by this PR; the falsifier is the live twin above.

Contract

contracts/apr-parity-per-op-v1.yaml (kind: pattern; POP-OB-001..003 ↔ POP-F-001..003) — pv validate through scripts/pv_bin.sh: valid.

Decomposition (hook-clean, verified)

forward_single_with_cache 26/55 → 6/9 and single_cache_ffn_block 21/69 → 6/12 (cyclomatic/cognitive), split into small helpers. Oracle: the per-op CPU dump tree over 78 positions — 24,258 files bitwise identical before/after; GPU tree identical; same first diverging op.

Quorum

.pr/L0-1b/quorum.md — three families (gemini-3.1-pro, claude-sonnet-4-6, gpt-oss-120b): 3/3 endorse fixing the CPU side; lane 2's dissent ("per-32 matches the GPU" is falsified by the probe table) folded; the delegate's zero-code arm DIRECT_FP32_GEMV=1 measured 0.9508 → 0.999896 before the fix was written.

Receipt

docs/audits/impl-PMAT-1070-receipt.md — status: partial (step 2 open).

Writes

code: crates/apr-cli/src/commands/{parity_per_op.rs,parity_per_op_table.rs,mod.rs,dispatch_analysis.rs}, crates/apr-cli/src/extended_commands.rs, crates/aprender-serve/src/quantize/{mod.rs,direct_f32.rs,fused_gate_up.rs}, crates/aprender-serve/src/inference_trace/gpu_stage_dump.rs (+gpu_stage_dump/per_op_tap.rs), crates/aprender-serve/src/cuda/executor/{stage_dump.rs,mod.rs,layers/phase_attention.rs,layers/indexed_ffn.rs}, crates/aprender-serve/src/gguf/cuda/{mod.rs,mod_parity_gate.rs}, crates/aprender-serve/src/gguf/inference/forward/ffn_block.rs · contracts: contracts/apr-parity-per-op-v1.yaml · receipt · docs/audits/l0-1b-arms.md · .pr/L0-1b/** (plan, accept.sh, step0 evidence). No shared files.

Fleet pin note: this branch sits on agent/L0-1 (#3026) and carries its cuda-feature fix (gpu_profile_or_none error type).

🤖 Generated with Claude Code

[noahgift]

Rebased onto the new main (L0-1a landed), acceptance re-run 6/6

#3026 merged, so this branch merged origin/main. Three conflicts, all the same shape and all resolved to ours, because this is the branch where L0-1a's parity-gate API grew rather than a divergent take on it:

file	main	here
commands/mod.rs	no parity_per_op*	L0-1b's two modules
cuda/mod.rs	ParityGateRecord bool form	skipped_for_diagnosis(), load_time_skip() -> Option<Self>
mod_parity_gate.rs	admit_by_parity_gate(bool)	admit_by_parity_gate(Option<ParityGateRecord>)

"Ours" is a claim until the callers agree, so both crates were compiled after the resolution rather than eyeballed: cargo check -p aprender-serve --lib and cargo check -p apr-cli --lib, both clean.

.pr/L0-1b/accept.sh — 6/6, hardware legs included:

ok A1 cargo test -p apr-cli --lib parity_per_op_table
ok A2 cargo test -p aprender-serve --lib per_op_tap
ok A3 pv validate contracts/apr-parity-per-op-v1.yaml
ok A4 docs/audits/l0-1b-arms.md non-empty
ok A5 apr parity <1.5B> --per-op  → no diverging op, lm_head ≥ 0.98
ok A6 apr parity <7B>   --per-op  → no diverging op

Shared guards run pre-emptively rather than discovered one CI cycle at a time — claims ratchet, hardcoded paths (--full), competing harnesses, baseline ratchets, tree-reader registry: all 0.

Why this is the release

D-14 rescoped 0.66.0 to the CUDA parity fix and nothing else, and this PR is the fix. With it, C14 — the release criterion for 0.66's single claim — runs live and passes on lambda:

PASS qwen2.5-coder-1.5b-instruct: 78 positions, min cosine 0.9998 at position 36 >= 0.98
PASS qwen2.5-coder-0.5b-instruct: 78 positions, min cosine 0.9996 at position 0
PASS qwen2.5-coder-7b-instruct:   78 positions, min cosine 0.9996 at position 22
C14: measured=3 rc=0

Receipt: evidence/release/0.66.0/C14-live-lambda.md. The gx10 twin of the fix (GB10, sm_121, aarch64) is in evidence/parity/l0-1b/gx10/: 1.5B 0.950611 → 0.999583, 7B 0.998465 → 0.999782, first_divergence none on both.