PP-26 witness RED on gx10 (GB10 sm_121, CUDA 13.3): c=16 batch loses slot invariance at token 31; c=4 and lambda sm_89 stay invariant to 128

[noahgift]

Summary

The PP-26 batch-invariance witness (scripts/perf041_batched_parity_probe.sh, v3.1) is RED on gx10 (GB10, sm_121, CUDA 13.3, driver 590.48.01) at the v0.66.0 commit 53dd489e4, and green on lambda (RTX 4090, sm_89) at the same probe version. cuda-nightly run 34445264201 (2026-09-10T06:37Z), artifact pp26-witness-blackwell-gb10:

band	m formed	result	intra-batch agree to	m=1 agree to	reason
c=1	1	PASS	128	128	—
c=4	4	PASS	128	6	—
c=8	0	UNMEASURABLE	—	—	no batch with m>1 formed in the band's window
c=16	16	FAIL	31	6	slots of one batch disagree at chunk 31 < declared_min=64: the batch is not invariant across its own slots

Same probe on lambda (evidence/perf041/lambda/witness.json, c554a7731): c=1/4/8/16 all PASS, intra-batch agreement 128/128 in every band.

Model qwen2.5-coder-1.5b-instruct-q4_k_m.gguf (sha256 cc324af0…), greedy, seed 0, n_predict 128, CUDA_BATCH_WINDOW_MS=200; binary sha256 a89cd992… built on the host.

Why this is a code-class finding, not env

• c=4 with m=4 is invariant to 128 on the same host, same binary, same model: the batched path itself works on sm_121.
• c=16 with m=16 loses slot invariance at token 31 — a batch-size-dependent divergence inside one batch (every slot receives the identical prompt). On sm_89 m=16 stays invariant. The candidates are the m≥8/16 tile or reduction path (a kernel-family threshold that lambda's sm_89 does not cross, or the sm_121 build of the same kernel), not the scheduler.
• The m=1-vs-batched divergence at token 6 is the known kernel-family divergence (recorded, not gated — PP-26 v3.1).

Ask

1. Reproduce on gx10 with the probe pinned to c=16 and vary m ∈ {8, 12, 16} to find the smallest m that breaks slot invariance; then bisect the kernel family (APR_DECODE_GEMM, CUBLAS_GEMM_THRESHOLD, FUSED_GATE_UP env knobs are recorded null in the witness).
2. Fix in the kernel, add the falsifier to the cuda suite that feat(PMAT-1095): 0.67 CUDA Rust fleet readiness — fix the two wrong-host GPU tests, add scripts/cuda_rust_fleet_check.sh with per-host receipts (gx10 PASS, yoga PASS, lambda-vector blocked on a reboot) #3068 makes green on gx10.
3. Until fixed, PP-26 on gx10 stays RED in cuda-nightly; the 0.67.0 row 67-F1 ("PP-26 witness green on gx10") cannot close by observation. Recommendation: 67-F1 slips to 0.68.0 with this ticket as its blocker (the F rows are the schedule's "slip first" class); the release-host witness for 0.67.0 is taken on lambda as for 0.66.0.

Also in that job

The byte-compare step (PP-26 - witness byte-compare against the previous run (BSE-11)) fails with "gh run list failed (auth/API error)" — gx10 has no gh (#3083). That is a separate tooling fix (REST with the job token), PR follows; it does not change the witness verdict above.

Evidence: run 34445264201, artifact 10139689077 (witness.json, marker.json status DEFECT, probe.log).

[noahgift]

slipped_from: 0.67.0 → 0.68.0 (freeze 2026-09-12T06:00Z). 67-F1 (#3097, the REST byte-compare) is parked behind this witness; the c=16 slot-invariance loss at token 31 on GB10 sm_121 is unresolved and is not a release blocker for 0.67.0 (decode parity gates run at c=1..4).