ci(cuda-nightly): 67-F1 — PP-26 byte-compare via REST with the job token; gx10 has no gh (#3096 filed for the witness itself) by noahgift · Pull Request #3097 · paiml/aprender

noahgift · 2026-09-10T13:53:46Z

67-F1 (PMAT-1098, epic #3078) — cuda-nightly's PP-26 byte-compare read the previous witness through `gh`, which no CUDA box has

Every cuda-nightly on gx10 ended the falsifiers job with PP-26 witness: gh run list failed (auth/API error) — that is gh: command not found wearing an auth error's label (#3083 / 67-C2 measured the missing toolset; #3074 already moved binary-release.yml to REST for the same reason).

Change (one step, .github/workflows/cuda-nightly.yml): the byte-compare step lists the last successful cuda-nightly run and its pp26-witness-<leg> artifact through the REST API with the job token (curl + python3, both present on gx10 and yoga), downloads and unzips it. The three outcomes stay distinct exactly as before: a REST/auth error is exit 1 (never a silent NO-BASELINE); "no such artifact on the last green run" is NO-BASELINE; a download/unzip failure is exit 1 with the error on stderr. Step-level env: GITHUB_TOKEN: ${{ github.token }}.

Not changed by this PR: the witness verdict itself. Run 34445264201's witness on gx10 is a code-class RED — at c=16 a batch loses slot invariance at token 31 on GB10 while c=4 on the same host and every band on the 4090 stay invariant to 128. Filed as #3096 with the per-band table; recommendation there: 67-F1's "witness green on gx10" slips to 0.68.0 behind #3096.

Acceptance (re-run by the orchestrator on 7167443): YAML parses; check_runner_labels.sh ✓; check_guards_are_wired.sh PASS (ratcheted); make gate rc=0 (f1_gate.log). No hosted runner added. Receipt stays status: partial per G-11.

Pmat-Ticket: PMAT-1098

🤖 Generated with Claude Code

…tness through gh, which no CUDA box has; REST with the job token + python3, three outcomes kept distinct (#3096, PMAT-1098) Every nightly on gx10 ended in 'gh run list failed (auth/API error)' — gh: command not found wearing an auth error's label (#3083). The witness verdict itself (c=16 slot invariance lost at token 31 on GB10) is #3096 and is not changed by this step. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

github-actions · 2026-09-10T14:24:14Z

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3097 head=8ef244974465cf522ec2abc304dd25a41e9753fc verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

…oint in the workspace — pre-create it as the runner (rebuild 34488955316 died at mkdir after a green 1m54s build) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…rst 'Host layout' step exports CI_TARGETS_ROOT / CI_CARGO_ROOT / SCCACHE_HOST_DIR / CI_REGISTRY / IMAGE with the intel clean-room defaults, 33 hardcoded sites read them; byte-identical on intel (#3100) Also: check_runner_labels.sh accepts the `build` pool label. No job changes where it runs yet (BP-3). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…asset-target-mountpoint

… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…8-67-F1-pp26-bytecompare-rest

… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…red review job held yoga's only pool slot for 15 min while required jobs queued (#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… so gx10-build's idle cores may take it (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…too (runs_on input, paiml/.github#67; operator: move now) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…8-67-F1-pp26-bytecompare-rest

…tpoint' into PMAT-1098-build-pool-any-of-three # Conflicts: # .github/workflows/ci.yml

…pool; guard-tree/guard-cargo/mutants/vendored-schemas/pr-review-* and the reusable ci jobs back on clean-room until measured (ci / security refuses to run without pmat, absent on yoga; #3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…8-67-F1-pp26-bytecompare-rest

…ls (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…n the checkout is too shallow to hold it — intel passed on leftover history, every fresh yoga checkout died 'invalid object name 42be156' 42be156 is on main (the v1.5 spec's squash-merge, 2026-09-05) but deeper than CI's checkout; the long-lived intel workspaces carried it from earlier fetches, the ephemeral yoga runners never do, so guard-tree's 'PP-066 spec v1.6 defect table and its v1.5 RED proof' step exited 128 there (#3097 on yoga-build3). One object fetched by SHA when absent; the table and its RED proof are unchanged. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit fecb358)

noahgift · 2026-09-11T19:35:03Z

Dequeued + auto-merge disarmed by the fleet loop (2026-09-11 ~19:40Z): gx10 pool1/pool2/build now carry clean-room (operator P0: gx10 never idle next to an aprender queue) and this branch does not carry the #3112 aarch64 clippy fix — a merge-group ci / lint landing on gx10 would evict the entry and burn a queue cycle (the 08:20Z lesson). Re-arm after the 0.67 train (#3127) lands and this branch is update-branch'd. Separately, 67-F1's own witness #3096 is still RED on gx10, so F1 is expected to slip to 0.68 at the freeze (06x spec: scope slips, dates do not).

…and the merge queue mirrors the PR instead of paying an hour for a moved main (67-E1 + 67-E2, paiml#3084, PMAT-1098) (paiml#3089) * roadmap: mint PMAT-1098 — the 0.67.0 release train (epic paiml#3078) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BjhtNUSensCYpQb3mCYLod * test(ci-tier): RED — 8 rows pin the filterset translation the quick tier does not do yet The quick tier turns 41 tree-reader targets into a 26-invocation `&&` chain (one cargo per crate, serial, 26 compiles of the shared deps): 55 min on a one-file YAML PR, timed out at 60 under fleet load (paiml#3070). The replacement is ONE build graph filtered by a nextest filterset, so the translation from the registry tokens to that expression is the new load-bearing step. These rows fail now (--filterset is not a flag yet) and pin BOTH polarities: each recognised token becomes exactly one clause, and an unrecognised token is ENV (exit 2) rather than a silently dropped target — a dropped token is a tree-reader test that stops running while the step stays green, which is the failure mode the registry exists to prevent. Row 22 counts clauses against registry lines so nothing can be dropped in bulk either. Row 21 was written fail-open first (`grep -q ":--" && LEFTOVER || NONE-LEFT` passes on EMPTY output) and is hardened here before it was ever green. Refs paiml#3084 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf(ci): the quick tier's 26 cargo invocations become ONE build graph GREEN for the rows added in the previous commit. `scripts/ci_test_tier.sh --filterset` turns the tree-reader registry's tokens into a cargo-nextest filterset — `(package(c) & kind(lib))`, `(package(c) & kind(bin))`, `binary_id(c::name)`, UNIONed with `|`. ci.yml's part-2 quick-tier step now runs ONE `cargo nextest run --profile ci --workspace --lib --tests --exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute -E "$EXPR"` instead of a `&&` chain of one cargo per crate. The binary-id forms are nextest's own and were VERIFIED on this workspace with cargo-nextest 0.9.132 (`cargo nextest list --message-format json`), not assumed: a lib suite's id is the bare package name, an integration target's is `package::target`, a bin's is `package::bin/name`. MEASURED on lambda, warm target dir (both listings from the same tree): one graph 65263 tests, 41 binaries, 39 suites with >=1 test build+list 96s, execution 220.2s, 316s wall, rc=0, 65263 passed 20-crate 65204 tests, 40 suites chain (20 distinct crates, not 26 — the 41 tokens group into 20) The two sets are NOT identical and the difference is entirely FEATURE UNIFICATION, never target selection: * A\B = 258 tests. Features the workspace resolve unifies ON and a per-package resolve leaves OFF (aprender-core hf_hub / inspect::safetensors / format::homomorphic / format::quantize, aprender-orchestrate mcp_json, aprender-test-lib async driver). * B\A = 199 tests, and ZERO of them exist under the workspace resolve at all — they are cfg(feature)-gated out. 197 are aprender-test-lib's browser mock suites, 2 are aprender-present-cli::gate_can_fail. So the FULL tier, which is `--workspace --lib`, never ran them either: the old chain was compiling a DIFFERENT aprender-test-lib than the tier it is supposed to approximate. Aligning the quick tier with the full tier's resolve is the fix, not a loss. * Suite selection is otherwise exact: 41 clauses -> nextest reports "Starting 65263 tests across 41 binaries". The two suites with no tests are aprender-core::setfit_conformance and ::falsification_spec_v10_tests, built and selected in BOTH but empty under default features (they are the full tier's --features setfit targets). * `--bins` maps to `kind(bin)`, not to the whole package. The old chain ran `cargo nextest run -p aprender-present-cli` with NO flags, so it also swept up `gate_can_fail` — an integration target that is in neither the registry nor the unwired ledger, i.e. not a tree reader at all. `kind(bin)` selects exactly what the token claims: the cfg(test) unit tests inside the bins of a crate that has no src/lib.rs (which is the only reason the token exists — `--lib` there is `error: no library targets found`). Both quick-tier steps drop to `timeout-minutes: 20`. 60 is the number that let this step burn 55 minutes on a one-file YAML PR (run 34449608126) and then die at the cap under fleet load on paiml#3063 (paiml#3070); the budget is the assertion. check_tree_reader_tests.sh is untouched and still derives the same 41 targets: its `full_tier_excludes` regex requires `--workspace --lib` followed IMMEDIATELY by ` --exclude`, and the new line has `--tests` in between, so it still reads the full tier's line and only that one (verified by running the regex). Refs paiml#3084 Closes paiml#3070 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(ci-tier): RED — 11 rows pin a queue that MIRRORS the PR instead of paying an hour for a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf(ci): the merge queue mirrors the PR instead of paying an hour for a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(release): the bullseye CUDA build left a root-owned target/ mountpoint in the workspace — pre-create it as the runner (rebuild 34488955316 died at mkdir after a green 1m54s build) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): BP-1 — the host layout is a property of the box: a first 'Host layout' step exports CI_TARGETS_ROOT / CI_CARGO_ROOT / SCCACHE_HOST_DIR / CI_REGISTRY / IMAGE with the intel clean-room defaults, 33 hardcoded sites read them; byte-identical on intel (paiml#3100) Also: check_runner_labels.sh accepts the `build` pool label. No job changes where it runs yet (BP-3). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): BP-3 — the containerized jobs and gate run on ANY x86 box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; paiml#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): BP-3 — the containerized jobs and gate run on ANY x86 box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; paiml#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): the pr-review-* jobs stay on clean-room — a non-required review job held yoga's only pool slot for 15 min while required jobs queued (paiml#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): vendored-schemas on the any-arch pool — pure scripts, so gx10-build's idle cores may take it (paiml#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): the reusable sovereign-ci jobs run on the build pool too (runs_on input, paiml/.github#67; operator: move now) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): measured-only routing — workspace-test + gate on the pool; guard-tree/guard-cargo/mutants/vendored-schemas/pr-review-* and the reusable ci jobs back on clean-room until measured (ci / security refuses to run without pmat, absent on yoga; paiml#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels (paiml#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ci): RUSTC_WRAPPER names sccache itself — the image's #!/bin/sh rustc-sccache wrapper runs under dash, which drops env vars whose names are not shell identifiers, so CARGO_BIN_EXE_aprender-explain never reached rustc The quick tier (one workspace build over --lib --tests) is the first CI step to compile aprender-explain's integration test, whose binary name has a hyphen: env!("CARGO_BIN_EXE_aprender-explain") failed 'not defined at compile time' on intel and yoga. Proven: through /bin/sh (dash) CARGO_BIN_EXE_aprender-explain is dropped and CARGO_BIN_EXE_apr survives, in the sovereign-ci image and locally; the step's exact build WITHOUT the wrapper compiled clean on yoga (12m42s). All 8 RUSTC_WRAPPER settings now name /usr/local/cargo/bin/sccache — the same binary the wrapper exec'd, minus the shell. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): pp066_v16_defects.sh --v15-red fetches the v1.5 commit when the checkout is too shallow to hold it — intel passed on leftover history, every fresh yoga checkout died 'invalid object name 42be156' 42be156 is on main (the v1.5 spec's squash-merge, 2026-09-05) but deeper than CI's checkout; the long-lived intel workspaces carried it from earlier fetches, the ephemeral yoga runners never do, so guard-tree's 'PP-066 spec v1.6 defect table and its v1.5 RED proof' step exited 128 there (paiml#3097 on yoga-build3). One object fetched by SHA when absent; the table and its RED proof are unchanged. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit fecb358) * fix(test): V15_SHA is the full object id — a remote cannot resolve an abbreviated one, so the shallow-checkout fetch fell through Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit ba69692) * ci(quick tier): the tree-reader step gets 45 minutes, since its measured runs never fit 20 This PR's quick-tier part 2 ("every test target that reads the tree") timed out at its own 20-minute cap twice under train load: run 34542045481 on yoga-build2 compiled 811 crates in ~9 min (a cold per-run target dir, with the shared sccache warm) and was part way through 65,266 tests at the cap, and attempt 2 on intel-clean-room-15 timed out at 20 minutes as well. The "~6 min" figure was never reproduced in CI. 45 minutes is still well under the serial tier's 60; the step is re-tightened once it is measured under load. No guard, test or spec pins the old 20. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…graph (PMAT-1098) (paiml#3127) * docs(spec): NVIDIA CUDA Rust integration for GPU quality stabilization (0.67) Research + two-pass review (agy grillme, agy /teamwork-preview) of NVIDIA's CUDA Rust announcement against aprender's GPU surface. Recommendation: 0.67 ships T0 + O2, neither of which needs an NVIDIA dependency. Everything requiring cuda-core/cutile defers to 0.68 behind a driver/toolkit prerequisite no fleet host meets today. Load-bearing findings, all measured on 2026-09-09 (HEAD 1faf09015): - G2/G3: aprender-gpu holds 2,620 tests and is excluded from every required check (ci.yml:417). 444 of them need no GPU and run in 0.10s. - G18: contracts/trueno/ptx-codegen-safety-v1.yaml has asserted a register/ occupancy budget since 2026-04-06, and it is vacuous — the generated macro contract_register_budget! is invoked NOWHERE in the tree, its postcondition names an unbound symbol, its falsifier is prose, and it carries registry: true. Neither review lane found this; it is why the scope is not reduced to T0 alone. - G7/G8/G9: cuda-core 0.3.1 builds on stable Rust with CUDA 12.x but refuses at runtime on driver 570.207, and passes end-to-end on gx10 (sm_121, CUDA 13.0, driver 590.48.01) through load_module_from_ptx_src -> launch. - G12: cutile needs CUDA 13.1+/13.2+; no fleet host qualifies. The draft's "upgrade gx10" step is withdrawn — gx10 is the only Blackwell host and the only cuda-nightly host. Evaluation moves to yoga. - G16/G20: an oracle crate must live in experiments/ with its own [workspace]; --all-features (Makefile:659,994,1010) would otherwise hard-fail make coverage and make mutants for contributors without a CUDA 13 toolkit. Every citation re-verified by script; two line numbers taken from a subagent report were wrong (206/293 vs the measured 207/294) and are corrected, with the lesson recorded in the review record. Refs: https://developer.nvidia.com/blog/introducing-cuda-rust-two-tracks-for-writing-gpu-kernels/ Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * docs(spec): operator ordered the gx10 CUDA 13.3 upgrade — executed, T3 unblocked Operator ruling 2026-09-09, verbatim: "false YOU WILL UPGRADE". This overrules the grill's RANK3 and the spec's own §7 recommendation. Per repo doctrine a review lane may not reopen an operator decision. Executed on gx10-a5b5: cuda-toolkit-13-3 (13.3.1-1, sbsa/arm64, 66 packages). No kernel-driver package (asserted driver-free before apt ran); driver unchanged at 590.48.01; CUDA 13.0 left on disk and intact. Result — the T3 blocker is gone (G21): cutile-rs 0.3.1 BUILDS AND RUNS on GB10 sm_121, stable rustc 1.95, 18.13s, JIT through CUDA Tile IR, len=1024 mismatches=0. Before the upgrade the fleet max was 13.0, below cutile's 13.1 floor, and this was untestable. T3 therefore moves into 0.67. Corrections this commit also carries: - G22 / §9.4: the upgrade was FIRST REPORTED HERE as "side-by-side, CI lane untouched". That was WRONG. /usr/local/cuda is a Debian ALTERNATIVES link, not a plain symlink; cuda-toolkit-13-3 registers priority 133 vs 13.0's 130, so the default flipped to 13.3 at 11:32:57 (/var/log/alternatives.log). The check that produced the false green was [ "$(readlink -f ...)" = "$(cat ...)" ]; chk "msg $(cat ...)" $? where the command substitution in the MESSAGE argument consumes $? before it is expanded, so $? carried cat's 0, not the test's 1. Reproduced both ways. Generalises the CLAUDE.md "never read $? through a pipe" rule: capture the status into a variable before any other substitution on the line. - §9.5: the post-upgrade regression gate was RUN, with a control. 2568 passed / 2 failed. Rolling the alternative back to 13.0 and re-running just those two shows 13.3 caused NEITHER: * test_alloc_oversize_100gb — FAILS on 13.0 AND 13.3 (pre-existing) * test_cublas_gemm_f16_... — PASSES alone on both; failed only inside the full suite (load-contended wall-clock assertion) Two pre-existing defects surfaced, both wrong-host assumptions: adversarial.rs:36 hard-codes "impossible on RTX 4090" (gx10 is a GB10 with unified memory, so the 100GB alloc legitimately succeeds), and cublas_tests.rs:174 asserts >50 TFLOP/s — a wall-clock assertion, the class already burned four times. Neither was visible because cuda-nightly.yml:242 runs only the perf053 filter, so the --features cuda suite has never been green on the Blackwell host and nothing said so. Rollback remains one command, no download: sudo update-alternatives --set cuda /usr/local/cuda-13.0 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * docs(spec): retarget to 0.68 and link tracking issue #3062 Operator direction: this spec ships in 0.68, not 0.67, and gets a GitHub ticket linked to the document. - Release targets shifted: the three shipping tiers (T0 un-dark the GPU crate, O2 arm the vacuous register_budget contract, T3 cutile A/B on gx10) move 0.67 -> 0.68; the two deferred tiers (T1 cuda-core loader differential, T2 cuda-oxide sanitizer + schedule-fuzzing) move 0.68 -> 0.69. Shifted in that order so the deferred tier did not collide with the shipping one. - Header now carries the bidirectional link: tracking issue #3062 (milestone 0.68.0) and spec PR #3061. Issue #3062 cites this document as the spec of record and carries the acceptance checklist. - Milestone 0.68.0 created (repo had 0.66.0 and 0.67.0 only). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * docs(spec): this spec IS 0.67; the previous 0.67 queue slips to 0.68 Operator direction: the CUDA Rust work is the 0.67 release, and everything previously targeted at 0.67 moves to 0.68. Spec: - Shipping tiers (T0 un-dark the GPU crate, O2 arm the vacuous register_budget contract, T3 cutile A/B on gx10) target 0.67. - Deferred tiers (T1 cuda-core loader differential, T2 cuda-oxide sanitizer + schedule-fuzzing) target 0.68. - Shifted 0.68->0.67 first, then 0.69->0.68, so the deferred tier could not collide with the shipping one. GitHub, every mutation read back from the API rather than assumed: - 16 open issues moved 0.67.0 -> 0.68.0 (#3045 #3024 #3019 #3018 #3013 #3002 #2910 #2909 #2908 #2906 #2905 #2904 #2893 #2891 #2890 #2869). The milestone held 0 closed issues, so no completed work was retargeted. - EPIC #3062 and this PR now sit on 0.67.0. Counts read back: 0.67.0 open=2, 0.68.0 open=16. - Milestone descriptions corrected: 0.67.0 is the CUDA Rust release; 0.68.0 is the PP-066 carry-over plus the deferred T1/T2 tiers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * spec(release): 06x release schedule — 0.67.0→0.70.0 every 2–3 days, one epic per train, priorities A–G as rows with acceptance commands; contract + drift gate (PMAT-1097) Every figure in §0 carries the command that measured it; comparator ratios sit beside their evidence/ receipt (the claim guards pass). The drift gate FALSIFY-DOCS-CLAUDE-001 now names the schedule (observed RED on a bogus path, exit 101; GREEN after revert). contracts/release-schedule-06x-v1.yaml validates; falsifiers 001-008 pass at this HEAD (003 was RED until the four epics #3078-#3081 existed). Epics: #3078 0.67.0 · #3079 0.68.0 · #3080 0.69.0 · #3081 0.70.0. P0 five-whys: #3082 #3083 #3084. Milestones 0.69.0 (#6) and 0.70.0 (#7) created. Operator 2026-09-10: 'override "kind:docs" block if needed' — filed kind:code (contract + test ride with the spec). Pmat-Ticket: PMAT-1097 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JZjEDVTFkFv71yCPD1dmUJ * spec(release): fold in the agy /teamwork-preview review — six findings re-run, five accepted, one premise refuted (PMAT-1097) F1 runner group gpu-x86 re-measured (allows public repos, no workflow restriction) — 67-D0 tightened. F2 milestones 0.69.0/0.70.0 created by this ticket. F3 premise refuted against crates/apr-cli/Cargo.toml line 75/89 (cuda is opt-in); smoke-cpu remedy taken. F4 accepted with a bounded remedy: the queue mirrors the PR tier, over-cap PRs add cargo check --workspace, root-manifest PRs keep FULL, residual named in R2. F5 rescoping of 'default' stated at the point of use. F6 near-tie vs defect rule for the PP-26 witness. §10 records provenance: one lane, fan-out unmeasured, --sandbox deviation declared, isolation byte-identical. Pmat-Ticket: PMAT-1097 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JZjEDVTFkFv71yCPD1dmUJ * audit(PMAT-1097): impl receipt and estimates row Pmat-Ticket: PMAT-1097 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JZjEDVTFkFv71yCPD1dmUJ * spec(release): same-day amendment — disposable Docker runners (infra#500) and the containerised CUDA asset build (#3086) enter G7, 67-A1, §2 C and 67-D0 (PMAT-1097) Pmat-Ticket: PMAT-1097 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JZjEDVTFkFv71yCPD1dmUJ * roadmap: mint PMAT-1098 — the 0.67.0 release train (epic #3078) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BjhtNUSensCYpQb3mCYLod * test(ci-tier): RED — 8 rows pin the filterset translation the quick tier does not do yet The quick tier turns 41 tree-reader targets into a 26-invocation `&&` chain (one cargo per crate, serial, 26 compiles of the shared deps): 55 min on a one-file YAML PR, timed out at 60 under fleet load (#3070). The replacement is ONE build graph filtered by a nextest filterset, so the translation from the registry tokens to that expression is the new load-bearing step. These rows fail now (--filterset is not a flag yet) and pin BOTH polarities: each recognised token becomes exactly one clause, and an unrecognised token is ENV (exit 2) rather than a silently dropped target — a dropped token is a tree-reader test that stops running while the step stays green, which is the failure mode the registry exists to prevent. Row 22 counts clauses against registry lines so nothing can be dropped in bulk either. Row 21 was written fail-open first (`grep -q ":--" && LEFTOVER || NONE-LEFT` passes on EMPTY output) and is hardened here before it was ever green. Refs #3084 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf(ci): the quick tier's 26 cargo invocations become ONE build graph GREEN for the rows added in the previous commit. `scripts/ci_test_tier.sh --filterset` turns the tree-reader registry's tokens into a cargo-nextest filterset — `(package(c) & kind(lib))`, `(package(c) & kind(bin))`, `binary_id(c::name)`, UNIONed with `|`. ci.yml's part-2 quick-tier step now runs ONE `cargo nextest run --profile ci --workspace --lib --tests --exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute -E "$EXPR"` instead of a `&&` chain of one cargo per crate. The binary-id forms are nextest's own and were VERIFIED on this workspace with cargo-nextest 0.9.132 (`cargo nextest list --message-format json`), not assumed: a lib suite's id is the bare package name, an integration target's is `package::target`, a bin's is `package::bin/name`. MEASURED on lambda, warm target dir (both listings from the same tree): one graph 65263 tests, 41 binaries, 39 suites with >=1 test build+list 96s, execution 220.2s, 316s wall, rc=0, 65263 passed 20-crate 65204 tests, 40 suites chain (20 distinct crates, not 26 — the 41 tokens group into 20) The two sets are NOT identical and the difference is entirely FEATURE UNIFICATION, never target selection: * A\B = 258 tests. Features the workspace resolve unifies ON and a per-package resolve leaves OFF (aprender-core hf_hub / inspect::safetensors / format::homomorphic / format::quantize, aprender-orchestrate mcp_json, aprender-test-lib async driver). * B\A = 199 tests, and ZERO of them exist under the workspace resolve at all — they are cfg(feature)-gated out. 197 are aprender-test-lib's browser mock suites, 2 are aprender-present-cli::gate_can_fail. So the FULL tier, which is `--workspace --lib`, never ran them either: the old chain was compiling a DIFFERENT aprender-test-lib than the tier it is supposed to approximate. Aligning the quick tier with the full tier's resolve is the fix, not a loss. * Suite selection is otherwise exact: 41 clauses -> nextest reports "Starting 65263 tests across 41 binaries". The two suites with no tests are aprender-core::setfit_conformance and ::falsification_spec_v10_tests, built and selected in BOTH but empty under default features (they are the full tier's --features setfit targets). * `--bins` maps to `kind(bin)`, not to the whole package. The old chain ran `cargo nextest run -p aprender-present-cli` with NO flags, so it also swept up `gate_can_fail` — an integration target that is in neither the registry nor the unwired ledger, i.e. not a tree reader at all. `kind(bin)` selects exactly what the token claims: the cfg(test) unit tests inside the bins of a crate that has no src/lib.rs (which is the only reason the token exists — `--lib` there is `error: no library targets found`). Both quick-tier steps drop to `timeout-minutes: 20`. 60 is the number that let this step burn 55 minutes on a one-file YAML PR (run 34449608126) and then die at the cap under fleet load on #3063 (#3070); the budget is the assertion. check_tree_reader_tests.sh is untouched and still derives the same 41 targets: its `full_tier_excludes` regex requires `--workspace --lib` followed IMMEDIATELY by ` --exclude`, and the new line has `--tests` in between, so it still reads the full tier's line and only that one (verified by running the regex). Refs #3084 Closes #3070 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(67-C2): the falsifier for a preflight that does not exist yet — RED at row 0 Ten rows, both polarities: a scratch PATH with every default tool is GREEN and the same PATH minus jq is RED naming MISSING jq; the JSON side-channel parses and records the missing tool; a usage error is exit 2 and a missing tool is exit 1; and the wiring itself is checked — every self-hosted job in fleet-toolset.yml, binary-release.yml and cuda-nightly.yml must run the preflight immediately after checkout, mutation-verified by deleting the step from a workflow copy. Run 34448908554 built a 21 MB CUDA asset on gx10 and died on the upload with `gh: command not found`. Nothing checked the toolset a workflow assumes. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(67-C2): name the missing tool at second zero, not after the GPU build scripts/ci_self_hosted_preflight.sh — defaults jq curl git python3 tar sha256sum rustup cargo; --cuda adds nvidia-smi AND asserts a device is actually visible; --need takes the extras a job knows it wants. `gh` is deliberately NOT a default: the CUDA lane moved to REST with curl+python3 because fleet boxes have no gh (#3074), and defaulting it would re-assert the assumption that failed. Exit 1 is a finding about the BOX, exit 2 a finding about the STEP that called this — merged codes send the wrong person. Identity (runner/labels/arch/glibc, driver under --cuda) is printed and, when PREFLIGHT_OUT or RUNNER_TEMP is set, written as JSON: that file is the fleet probe output. 13-row hermetic case table, both polarities, every tool a stub in a scratch PATH so the verdict does not depend on this box. Two rows were RED for the wrong reason first and are recorded as comments: the script called dirname/basename (now parameter expansion) and the nvidia-smi stub catted a fixture, so --cuda reported zero devices because `cat` was not on the scratch PATH. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(67-C2): every self-hosted job asks the box what it has, before it spends an hour The preflight is now the step immediately after checkout in all seven self-hosted jobs: binary-release.yml build-apr-cuda (--cuda --need docker objdump), smoke-cuda (same; both GPU boxes carry `docker` as a runner LABEL, so a pool that stopped matching its own labels is worth one second), and the two clean-room jobs verify-cuda-assets and summary, which had no checkout at all; plus the gx10 lane in cuda-nightly.yml. Not before `decide` and not before the checkout there: on a yield night the box has no working tree, and a step that bashes a repo path would exit 127 and turn a deliberately-yielded night RED. fleet-toolset.yml probes all three pools daily at 04:47 UTC and uploads toolset-<label>/preflight.json, 90-day retention, plus one table in the run summary. ARTIFACT + SUMMARY, NOT A PR: a PR per day against evidence/ is ~365 machine-written PRs a year through a queue that runs at ~1 PR/hour, so the cost of the record would exceed the record. A zero-row roll-up exits 1 — no preflight.json downloaded is a broken probe, not a clean fleet. The falsifier built its universe from the wrong side and said so: `runs-on` alone saw 3 jobs, missing build-apr-cuda and smoke-cuda, whose selector is `${{ fromJSON(matrix.labels) }}` — the two GPU jobs this row exists for. It now reads the strategy block too, and sees 7. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(ci-tier): RED — 11 rows pin a queue that MIRRORS the PR instead of paying an hour for a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf(ci): the merge queue mirrors the PR instead of paying an hour for a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(release): RED — nothing asserts the four apr assets on a tag Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(release): one checker for the sixteen assets a tag owes — v0.66.0 is missing four Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(release): the CPU half of the four-asset rule — build, verify and smoke it Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(release): C13 reads the tag from the manifest and can fail — both polarities in the table Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(release): the C13 fixture cleanup was an unguarded rm -rf (bashrs SEC011) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(dogfood): the post-publish phase asks whether the tag got its four apr binaries Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(guard-tree): RED — six rows pin a dispatcher that runs its guards concurrently, deterministically and fail-closed Row (e) is the discriminator and it reads no clock: two planted guards rendezvous through marker files, so a serial dispatcher fails because the peer never started. Rows (a)-(d),(f) pin the universe, the named failure, the count, byte-identical ordering across two runs, and a guard that dies without a verdict counting as FAILED. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf(ci): the guard jobs pay 71 minutes for a tree they were asked to judge in 20 Measured on run 34449608126 (PR #3074): guard-cargo 43.8 min over 80 steps, guard-tree 27.3 min over 45, and 18.4 of guard-tree's minutes were ONE step running ~35 independent guards one at a time on a 48-core box. That step now dispatches through an xargs pool (bounded -P 8, the clean-room pool is 16 runners on one host): 289 s -> 96 s locally on the real universe, same 41 checks. Seven tree-universe steps (26.8 min: the tier case table, model-tests, book examples, aprender-profile, the fd-0 scan, publish safety, wasm32) move to guards-nightly.yml with their name, run block and docker/mount shape verbatim, so check_guards_are_wired.sh -- which greps every workflow file, now pinned by rows 5-7 of its own case table -- still finds them wired. Both jobs take timeout-minutes: 20, the operator's budget, which is SMALLER than BSE-05's answer and overrides it. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * guard(ratchets): classify scripts/guards_nightly_manifest.txt as a ledger, not a ratchet — the one line 67-E3's worker could not write (out of its scope) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(ci-gpu): RED — 20 rows pin a GPU selection that does not exist and two PR jobs no runner has ever seen Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(gpu): two advisory PR jobs that SKIP the GPU hosts they do not need — 67-C1 gx10, 67-D1 yoga Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf-matrix: declare yoga (rtx-4060-laptop, sm_89, ci_runner self-hosted) so a GPU-labelled job on it may carry concurrency perf-yoga — the one out-of-scope entry 67-C1/D1's worker could not write Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf-matrix: re-vendor crates/aprender-test-lib/perf-matrix.vendored.yaml after declaring yoga (build.rs refuses drift, PMAT-958) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(readme): regenerate the CONTRACT_COUNT block — the merge tree carries 1818 contracts (release-schedule-06x-v1.yaml); FALSIFY-README-002 is an equality on a generated number Pmat-Ticket: PMAT-1097 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * roadmap: PMAT-1100 — triage pass filing every open issue/PR into a release train (kind:triage) Pmat-Ticket: PMAT-1100 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(release): the bullseye CUDA build left a root-owned target/ mountpoint in the workspace — pre-create it as the runner (rebuild 34488955316 died at mkdir after a green 1m54s build) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(gguf): Qwen3.5/Qwen3.8 hybrid GGUFs — the refusal names BOTH backends and the CLI stops promising a CPU fallback (#3091 ask 2, #3090 related) unsupported_architecture_reason now says NEITHER the CPU nor the GPU backend implements Gated DeltaNet/SSM layers yet, names the triggering tensor and #3090/#3091; the chat/run CLI no longer prints 'will use CPU' for a format-level refusal and exits non-zero on the first attempt; apr qa reports it as a structured Capability Match failure; the qwen3_5 arch-constraints row describes the real hybrid. Measured on lambda with ~/models/Qwen3.5-0.8B-Q4_K_M.gguf: unsupported_architecture 3/3, chat_load 1/1, apr run rc=8 (message contains 'neither', no 'will use CPU'), apr qa structured FAIL, clippy 0, fmt 0, make gate rc=0. Lane receipt: achieved (agy goal lane; isolation violation rc=3 recorded — the lane committed in the shared checkout). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): BP-1 — the host layout is a property of the box: a first 'Host layout' step exports CI_TARGETS_ROOT / CI_CARGO_ROOT / SCCACHE_HOST_DIR / CI_REGISTRY / IMAGE with the intel clean-room defaults, 33 hardcoded sites read them; byte-identical on intel (#3100) Also: check_runner_labels.sh accepts the `build` pool label. No job changes where it runs yet (BP-3). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): BP-3 — the containerized jobs and gate run on ANY x86 box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(audits): PMAT-1100 triage receipt — 263 open issues/PRs filed into release trains (33/33 batches verified), Alfredo's first, 8 closes under the operator's words with 6 epic-child closes reversed Pmat-Ticket: PMAT-1100 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): BP-3 — the containerized jobs and gate run on ANY x86 box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): the pr-review-* jobs stay on clean-room — a non-required review job held yoga's only pool slot for 15 min while required jobs queued (#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): vendored-schemas on the any-arch pool — pure scripts, so gx10-build's idle cores may take it (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): the reusable sovereign-ci jobs run on the build pool too (runs_on input, paiml/.github#67; operator: move now) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): measured-only routing — workspace-test + gate on the pool; guard-tree/guard-cargo/mutants/vendored-schemas/pr-review-* and the reusable ci jobs back on clean-room until measured (ci / security refuses to run without pmat, absent on yoga; #3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels now (yoga mirrors the intel host layout); BP-1 moved guard-cargo's target dir out of its job env and guard_tree_test.sh's case table failed on every runner Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: restore guard-cargo's literal sccache path (the revert had left ${SCCACHE_HOST_DIR}, undefined in that job) — ci.yml now byte-identical to main Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: drop the build-pool ci.yml edits by 3-way merge (the branch's last pool-free ci.yml ⊕ main) — keeps this branch's own guard changes (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: arch-neutral CI jobs may run on any clean-room box (intel, yoga, gx10); workspace-test stays X64 for x86 SIMD coverage; perf benchmarks pin the intel box (#3100, operator P0) clean-room now spans three boxes (runner labels; runner groups keep them aprender-only). Dropping X64 from guard-tree, guard-cargo, vendored-schemas, mutants, gate and pr-review-* lets the arm64 GB10 take them; workspace-test keeps X64 so every PR still runs the x86 SIMD paths. beat-speed-nightly and nightly-bench must measure on the same hardware as their recorded baselines, so they now require the intel label — without it yoga's different CPU would silently write wrong baselines. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: perf benchmarks require the intel label — clean-room now spans three boxes and yoga's CPU would silently record wrong baselines (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: drop the build-pool ci.yml edits (last pool-free ci.yml ⊕ main) and give guard-cargo's remaining cargo steps the job's own CARGO_HOME — moving model-tests to the nightly had left guard-cargo with none, and guard_tree_test's case table requires it Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: drop the build-pool ci.yml edits (last pool-free ci.yml ⊕ main) and give guard-cargo's remaining cargo steps the job's own CARGO_HOME — moving model-tests to the nightly had left guard-cargo with none, and guard_tree_test's case table requires it Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(format): the Qwen3.5 family YAML could not load — AttentionType learns hybrid_gated_deltanet, so the metadata-bounds falsifiers read it and the refusal can name the family #3099 marked contracts/model-families/qwen3_5.yaml honestly (attention_type: hybrid_gated_deltanet, is_supported_cpu/gpu: false), but AttentionType only parsed mha/gqa/mqa/ssm/linear, so FALSIFY-MB-003/004 panicked loading it on intel (workspace-test). New variant HybridGatedDeltaNet: parser, Display, the two exhaustive apr-cli matches (oracle kernel line, config.json explanation), and a parse assertion. aprender-core model_family + metadata_bounds: 104 passed; apr-cli check --tests clean. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ci): RUSTC_WRAPPER names sccache itself — the image's #!/bin/sh rustc-sccache wrapper runs under dash, which drops env vars whose names are not shell identifiers, so CARGO_BIN_EXE_aprender-explain never reached rustc The quick tier (one workspace build over --lib --tests) is the first CI step to compile aprender-explain's integration test, whose binary name has a hyphen: env!("CARGO_BIN_EXE_aprender-explain") failed 'not defined at compile time' on intel and yoga. Proven: through /bin/sh (dash) CARGO_BIN_EXE_aprender-explain is dropped and CARGO_BIN_EXE_apr survives, in the sovereign-ci image and locally; the step's exact build WITHOUT the wrapper compiled clean on yoga (12m42s). All 8 RUSTC_WRAPPER settings now name /usr/local/cargo/bin/sccache — the same binary the wrapper exec'd, minus the shell. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(release): the CPU apr assets build inside rust:1.93.0-bullseye like the cuda lane, and smoke-cpu GATES their glibc floor at GLIBC_2.31 — the floor variable it read was defined nowhere check_workflow_env_defined (guard-cargo) failed #3092: smoke-cpu interpolated APR_ASSET_GLIBC_FLOOR, which no job defines, so the floor check recorded a number and gated nothing. It could not honestly be set while build-apr-cpu built NATIVELY: the asset's floor was whatever the build box ran (0.66.0's native x86_64 asset needed GLIBC_2.43 and did not start on 22.04). build-apr-cpu now uses the cuda lane's bullseye sibling-container step verbatim minus --features cuda and the device lines (own cache dir, pre-created target mountpoint), and smoke-cpu declares the floor, so an asset importing a newer GLIBC_* symbol fails the release. Guards: check_workflow_env_defined OK (15 workflows), guard_tree_test 23/0, check_lockfile_current PASS. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): pp066_v16_defects.sh --v15-red fetches the v1.5 commit when the checkout is too shallow to hold it — intel passed on leftover history, every fresh yoga checkout died 'invalid object name 42be1560b' 42be1560b is on main (the v1.5 spec's squash-merge, 2026-09-05) but deeper than CI's checkout; the long-lived intel workspaces carried it from earlier fetches, the ephemeral yoga runners never do, so guard-tree's 'PP-066 spec v1.6 defect table and its v1.5 RED proof' step exited 128 there (#3097 on yoga-build3). One object fetched by SHA when absent; the table and its RED proof are unchanged. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): V15_SHA is the full object id — a remote cannot resolve an abbreviated one, so the shallow-checkout fetch fell through Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(guard): spec-conformance fixtures are bytes — the zero-width key rendered only under a UTF-8 locale The ledger_respend_zero_width_key fixture spelled its zero-width space as a \u200b escape and rendered it with printf '%b'. bash expands \u only when the locale is UTF-8; under a POSIX locale it keeps the six characters as literal text, so the key never collides with "lambda" and the row reads CLEAN. intel's runners run en_US.UTF-8, yoga's runner image ran POSIX, and main's guard-tree went RED on yoga-build3 (run 34529896596, job 103047791546: "BROKE ledger_respend_zero_width_key expected L1 got CLEAN"). The fixture now spells the bytes (\xe2\x80\x8b), which printf '%b' renders identically under any locale, and selftest() pins LC_ALL=C so a \u fixture fails on every box instead of only on the odd runner. Measured on a75e15460's tree: before, 40/40 under C.UTF-8 and 39/40 with no locale; after, 40/40 under both. Mutation: restoring the \u200b escape goes RED (39/40) under a C.UTF-8 ambient locale once the selftest pins C. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(test): pp066_v16_defects.sh --v15-red fetches the v1.5 commit when the checkout is too shallow to hold it — intel passed on leftover history, every fresh yoga checkout died 'invalid object name 42be1560b' 42be1560b is on main (the v1.5 spec's squash-merge, 2026-09-05) but deeper than CI's checkout; the long-lived intel workspaces carried it from earlier fetches, the ephemeral yoga runners never do, so guard-tree's 'PP-066 spec v1.6 defect table and its v1.5 RED proof' step exited 128 there (#3097 on yoga-build3). One object fetched by SHA when absent; the table and its RED proof are unchanged. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit fecb3581512b73bd95d4f52adde48e25354ab29c) * fix(test): V15_SHA is the full object id — a remote cannot resolve an abbreviated one, so the shallow-checkout fetch fell through Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit ba696920d912939d05ce969d8f72414e021452c9) * ci(release): the pv lane builds on the fleet — no GitHub-hosted runner left in binary-release.yml (#3073) binary-release.yml fires on `release: published`, and its pv job ran on ubuntu-latest (aarch64 via cross). Under the operator rule of 2026-09-10 ("WE DO NOT USE HOSTED GITHUB RUNNERS") a tag cut that way is not done. Each pv target now builds natively on the box of its architecture, x86_64 on yoga and aarch64 on gx10, inside rust:1.93.0-bullseye: the sibling-container pattern build-apr-cpu already uses. gnu assets carry a GLIBC_2.31 floor (cross's aarch64 image was 2.31 too; x86_64 came from ubuntu-latest's 2.39, so its floor drops), and musl assets are static, with musl-gcc named as their C compiler. New checks that can fail: the asset runs on its builder, a musl asset has no NEEDED entry, and a gnu asset imports nothing above GLIBC_2.31. Upload is the apr lanes' REST path, since these boxes carry no gh. Asset names are unchanged, so verify-apr-assets' eight pv names still hold. CROSS_VERSION, used only by the old pv job, is gone, and the header describes the new build. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(release): the musl pv lanes build on bookworm — bullseye-security's Release file has expired Rehearsal run 34543016302 (v0.67.0-rc.0): both gnu pv lanes built, both musl lanes died at `apt-get update` inside rust:1.93.0-bullseye, exit 100: "Release file for .../bullseye-security/ InRelease is expired". Only the musl lanes run apt (for musl-tools), which is why the gnu lanes on the same two boxes were green. The matrix now names each target's image: gnu stays on rust:1.93.0-bullseye (glibc 2.31 is its floor), musl moves to rust:1.93.0-bookworm, where apt still installs musl-tools. A static musl binary carries no builder glibc, so the floor promise is unchanged. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: no GitHub-hosted runner in nightly, book or book-contracts, and a guard that refuses one (#3073) Operator rule (2026-09-10, verbatim): "WE DO NOT USE HOSTED GITHUB RUNNERS. WE USE GX10 AND LAMBDA-LABS OR YOGA". Three workflows still ran on GitHub-hosted images: - nightly.yml: ubuntu-latest, ubuntu-24.04-arm, macos-latest x2, windows-latest. The Linux builds now run on the fleet (x86_64 on yoga, aarch64 natively on gx10) inside rust:1.93.0-bullseye, so the nightly's glibc floor drops from 2.39 to 2.31. check-activity and the release job run on the clean-room pool, and the release job deletes the old nightly over REST, because the fleet has no gh. No macOS or Windows self-hosted runner is registered, so those targets leave the nightly until one is. - book.yml (build + Pages deploy) and book-contracts.yml (six jobs) move to the clean-room pool. Both are path-filtered (0 and 1 runs in the last 24 h), so the pool barely notices. scripts/check_no_hosted_runners.sh refuses a hosted image token on any non-comment workflow line, not only on `runs-on:` lines: nightly.yml hid `ubuntu-latest` in a matrix value, which a runs-on-only scan never reads (and check_runner_labels.sh exempts hosted jobs by design). A shrink-only `file:job` baseline can excuse a job while its migration is in flight, and a stale entry is RED. The case table has 12 rows: both polarities, a stale-baseline row and an ENV row. Mutation: re-inserting `runs-on: ubuntu-latest` into nightly.yml turns the real-tree run RED (rc=1); restoring it turns it green. The guard is wired through guard_tree.sh's dispatch (check_guards_are_wired.sh PASS), and bashrs reports 0 errors. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(guard): the hosted-runner baseline is empty: every workflow runs on the fleet With the pv lane on gx10 + yoga (this branch), no workflow names a GitHub-hosted image, so the baseline check_no_hosted_runners.sh reads carries no entry. From here a hosted image anywhere is RED. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(book): self-hosted book jobs install their tools into the run's private root, not the shared home Moving book.yml onto the clean-room pool (#3073) put its three `cargo install` steps (bashrs, the local apr-cli, pmat) under check_cargo_install_private_root.sh, which guard-cargo ran red on yoga-eph (run 34544213071): a self-hosted `cargo install` into the shared ~/.cargo/bin replaces a binary another job on the same box may be exec'ing (aprender#2353). Each install now passes `--root "$RUNNER_TEMP/cargo-tools"`, and that root's bin is added to PATH last, so it resolves first. mdBook, which the guard does not cover, had the same race in ~/bin; it now unpacks into $RUNNER_TEMP/bin. Checked locally: check_cargo_install_private_root.sh OK (43 jobs, 9 installs, none shared) and its case table; check_no_hosted_runners, check_runner_labels, check_workflow_env_defined, check_workflow_path_filters and check_apr_bin_pinned all green. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(guard): drop the empty hosted-runner baseline, because an unclassified baseline file is RED check_baseline_ratchets.sh refuses any baseline file it cannot classify as a ratchet or as a stated exemption ("no rule is how a baseline arrives that nothing ever compares"). It turned guard-tree red on this PR for scripts/hosted_runner_baseline.txt, which had held nothing since the pv lane moved to the fleet. check_no_hosted_runners.sh already treats a missing baseline as zero exemptions, the strictest state, so the file goes. A later migration that needs an entry must add the file and classify it in the same change. Checked locally: check_baseline_ratchets, check_no_hosted_runners (real tree + 12-row case table) and check_guards_are_wired are all green. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(gpu): VRAM assertions hold each device to its own size, not every device to the RTX 4090's test_context_memory_info and test_context_total_memory asserted `total > 20 GB` ("RTX 4090 should have >20GB VRAM") on whatever device 0 is. yoga's RTX 4060 Laptop GPU has 8 GB, so this PR's cuda-unit job on yoga-eph failed both while saying nothing true about the card. A per-device floor table keyed by the CUDA device name now holds the 4090 to >20 GB (unchanged) and the RTX 4060 to >7 GB; an unlisted device (gx10's GB10 reports unified memory) gets the structural checks only. Proven on lambda's RTX 4090: both tests green, and a mutant raising the 4090 floor to 30 GB turns them RED (exit 101). cargo fmt --check clean; cargo deny and the aprender-contracts lib tests green. `cargo clippy -p aprender-gpu --features cuda --lib --tests -D warnings` reports 599 errors, none in this change: 551 are the banned unwrap() across the cuda test files, and the two in this file are the pre-existing line 13 (`assert!(... || true)`, overly_complex_bool_expr). CI does not lint these targets. The cuBLAS half of this job's failure ("cuBLAS library not found") was the runner containers' loader path, fixed on yoga and gx10 by LD_LIBRARY_PATH=/usr/local/cuda/lib64 (paiml/infra 4c2afe8). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(quick tier): the tree-reader step gets 45 minutes, since its measured runs never fit 20 This PR's quick-tier part 2 ("every test target that reads the tree") timed out at its own 20-minute cap twice under train load: run 34542045481 on yoga-build2 compiled 811 crates in ~9 min (a cold per-run target dir, with the shared sccache warm) and was part way through 65,266 tests at the cap, and attempt 2 on intel-clean-room-15 timed out at 20 minutes as well. The "~6 min" figure was never reproduced in CI. 45 minutes is still well under the serial tier's 60; the step is re-tightened once it is measured under load. No guard, test or spec pins the old 20. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(quantize): Q5_K readers decoded an invented layout — every Q5_K tensor on the CPU path and in apr import was wrong ggml's block_q5_K gives sub-blocks 2c and 2c+1 the same 32 qs bytes (low nibbles, then high nibbles) and takes the fifth bit of value l in sub-block s from bit s of qh[l]. realizar's dequantize_q5_k and fused_q5k_dot instead read the two nibbles of one byte as neighbouring values and took the fifth bit from qh[4s + l/8]. aprender-quant, the reader behind apr import, gave each sub-block its own 16 qs bytes, and its quantizer packed that way too. Measured on Qwen3.5-0.8B-Q4_K_M, whose attn_qkv and ssm_out are Q5_K: realizar disagreed with gguf-py on 5,959,751 of 6,291,456 values of blk.0.attn_qkv, and the layer-0 QKV sum was +26.64 against llama.cpp's -17.06. With this change it is -17.0967, and layers 0-2 of the #3091 trace match llama.cpp. Nothing caught it because every Q5_K test compared one aprender reader with another. realizar's dequantizer and fused dot now share a single block walker, so they cannot drift apart again. FALSIFY-QDOT-009 pins both readers to the 256 values that gguf-py (llama.cpp's own reader) produced from a block llama.cpp quantized. A round-trip on data whose sub-blocks all differ pins the packer to that reader; the old ramp round-trip gives every sub-block the same codes, and a packer that swapped sub-blocks survived it. Reverting either reader or the packer turns the tests red. The CUDA Q5_K gemv and gemm kernels carry two more wrong layouts. They get a separate PR. Refs #3091 Pmat-Ticket: PMAT-1101 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(compute,aarch64): gx10 could not run ci / lint — clippy errors only an ARM box sees, and #2567's parallel Q4_K path was never called The gx10-pool1 canary picked up #3089's ci / lint and failed it (job 103137079500). Lint is clean on x86. On aarch64, `cargo clippy --all-targets -- -D warnings` fails in aprender-compute and aprender-present-terminal, and so does the job's `-p aprender` fallback, because both crates are in the facade's dependency closure. - Four SIMD detectors end with `#[cfg(aarch64)] { return Neon; } Scalar`, which leaves the Scalar tail unreachable on ARM. Each tail is now a cfg'd block. - x86-only helpers were compiled on ARM and never called there: the Q6_K value extractor, the BLIS AVX-512 packers, the prefetch and NT-store thresholds, and the 32x6 packing import. Each now carries the x86_64 cfg of its only callers. - rounding.rs imported the NEON backend and never used it. - Five more detectors, in aprender-core, aprender-serve and trueno-ublk, had the same unreachable tail. They only showed up once the crates above them compiled. - Two x86_64-only BLIS packers in compute.rs are also dead on ARM. The cfg belongs on the functions, but compute.rs already carries 11 complexity violations, and the pre-commit gate refuses any edit to the file until it is decomposed. Until then, `mod compute;` carries a non-x86 `expect(dead_code)` with that reason. `expect` becomes an error as soon as the packers stop being dead. One of the dead functions was a real defect. #2567 made the non-x86 matmul_q4k_f32_parallel actually parallel (its doc records 1.21x on GB10), but the only call site sat inside the x86_64 block, so every ARM machine still ran the serial Q4_K kernel. The dispatcher now calls it on non-x86 above the same 8M-element threshold. A new test checks the threaded path against the serial one on every arch; the coverage module that exercised it only builds on x86_64. Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(cuda): the Q5_K GEMV read each value's fifth bit from a sequential bitmask — every Q5_K tensor decoded wrong on the GPU (#3111) In ggml's block_q5_K, the fifth bit of value l in sub-block s is bit s of qh[l], i.e. byte val_idx % 32, bit val_idx / 32. Q5KGemvKernel read byte val_idx / 8, bit val_idx % 8 instead. Its qs offset and nibble selection were already right, so only the high bit was scrambled. That was enough: on the FALSIFY-QDOT-009 golden block, the GPU dot against a ramp came out 2.659 where gguf-py gives 0.622. Measured on an RTX 4090 by running the new test against the shipped indexing. The only earlier GPU Q5_K test asserted `q5k_gemv(...).is_ok()` on synthetic weights. FALSIFY-QDOT-010 runs the kernel on the golden block from #3110, now a fixture shared via include!(). A one-hot activation must return each of the 256 gguf-py values, and a ramp must reproduce gguf-py's dot. Both tests pass with the fix and fail with the shipped indexing. Q5KKernel (the GEMM) has its own wrong layout: the qh and qs offsets and a 12-bit scale unpack. Nothing in inference dispatches it; only q5k_matvec in the layer-trace path and tests do. It stays tracked on #3111. Refs #3111 Pmat-Ticket: PMAT-1104 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(contract): FALSIFY-QDOT-009 becomes 007 — validate_contracts rejects gaps in falsification ids guard-cargo's "Contract corpus integrity" step (`cargo test -p aprender-contracts --test validate_contracts`) failed #3110 with "quantized-dot-product-v1: test ID gap: expected FALSIFY-QDOT-007, found FALSIFY-QDOT-009". I had skipped to 009 because crates/aprender-serve/src/quantize/contract_tests.rs already uses "FALSIFY-007" and "FALSIFY-QDOT-008". Those labels exist only in that file's panic strings. The corpus gate enforces the id sequence of the contract YAML itself, so the entry now takes the next id in the file, 007, everywhere it is cited. Pmat-Ticket: PMAT-1101 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(contract): FALSIFY-QDOT-010 becomes 008 — no gaps in the falsification ids Now that #3110's entry is FALSIFY-QDOT-007 (validate_contracts rejects gaps in the ids), the CUDA GEMV falsifier takes the next id, 008, and the shared fixture's header names 007/008. Pmat-Ticket: PMAT-1104 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(guard): the gemv doc's 1.21x ratio moved ten lines and the shrink-only baseline refused the re-key — the ratio had no receipt, so it leaves; the medians stay guard-cargo on #3112 failed check_perf_claims_cite_receipts.sh: the uncited 'speedup 1.21x' (PERF-010) sat at gemv/mod.rs:226 on main and this branch's ten inserted lines moved it to :236; a moved entry is growth under the line-keyed shrink-only ratchet. Re-keying is refused by design. The raw bench output was never preserved, so the honest remedy is the guard's second one: the ratio is deleted, the two medians remain, and the comment says why no ratio is stated. Baseline shrinks by one (235 -> 234). Both guards re-run green in this tree. Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * roadmap: PMAT-1106 — GPU-conditional tests must skip without an adapter (coverage nightly) Pmat-Ticket: PMAT-1106 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(compute): nine GPU tests panicked on a box without an adapter — the nightly coverage run was RED for six days on an environment fact backends::gpu::device::backward::tests called GpuDevice::new().expect("GPU device") nine times. Under cargo llvm-cov on yoga-build2 (a docker runner with no GPU passthrough) the two silu_backward tests panicked and the whole coverage run exited 2 (run 34575134766); the same class in aprender-cgp is #2848. A missing adapter is an environment fact, not a kernel defect: device_or_skip() prints 'SKIP: no GPU adapter on this host (<err>)' and the test returns. Proven both ways on lambda-vector: with VK_ICD_FILENAMES pointed at nothing, 9 SKIP lines and 9 passed in 0.00 s; with the real adapter, 0 SKIP lines and 9 passed in 1.42 s. The box that has an adapter stays the gate. Pmat-Ticket: PMAT-1106 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci: re-trigger — the push that merged #3089 in landed while the PR sat in the merge queue and GitHub started no pull_request run for it Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(test): falsification_measurement was dark for five months — cargo run needs --bin apr (apr-cli ships two binaries since 2026-04-18), and M008 asserts the exit code follows the printed verdict instead of assuming a jittered simulated run passes The E1 quick tier (#3089) is the first thing to run aprender-core's falsification_measurement_tests target on a PR: m001 died on 'cargo run could not determine which binary to run' (apr + apr-corpus-ingest), and m008 on 'CI thresholds not met' — cbtop --simulated draws jittered brick timings (3/7 bricks over budget, CV 88 %), so whether thresholds are met is a coin flip. Six cargo-run sites now name --bin apr; M008 asserts exit 0 <=> 'Status: PASS', the contract its name states. Both pass locally on lambda-vector. Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * roadmap: PMAT-3121 examples-dogfood (#3121) Pmat-Ticket: PMAT-3121 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * dogfood(release): CI compiled 981 examples and ran none — run them, and gate the cookbook and the release notes (#3121) Pmat-Ticket: PMAT-3121 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(examples-nightly): every workspace example builds and runs nightly on yoga, low priority; wires dogfood_examples.sh (#3121) Pmat-Ticket: PMAT-3121 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(test): falsify_cmp_003 read .clippy.toml from the CRATE dir — find_project_root stopped at the first Cargo.toml+src/, one level below the workspace; the target was dark in the full tier and the quick tier ran it first (#3112) Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(coverage-nightly): the low-priority coverage run lives on the yoga pool (never an intel PR slot), timeout 75 -> 150 min for yoga's core count; goal 95% (operator 2026-09-11) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(test): falsification_cuda_tests F062/F063 — a driver-present, zero-device container (yoga-eph nested CI docker) is an environment fact, not a defect; skip with a cited line instead of failing the quick tier (#3112) Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * guard(harnesses): dogfood_examples.sh is a dogfood LEDGER, not a competing harness — allowlisted with its reason (it records one run's wall seconds as provenance, states no rate) Pmat-Ticket: PMAT-3121 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * roadmap: PMAT-3124 triage pass — label the unlabeled open issues (#3124) Pmat-Ticket: PMAT-3124 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(test): falsify_bgn_002 wants the CRATE's build.rs (CARGO_MANIFEST_DIR), not the workspace root — the second path assumption in the same dark target; the whole target now runs green locally under nextest (116/116) Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * audit(triage): PMAT-3124 — 72 unlabeled open issues labeled (type + priority), 9 ledgers verified twice, 0 unlabeled remain (#3124) Pmat-Ticket: PMAT-3124 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(test): MUT-05/06/07 read .github/workflows/ci.yml relative to cwd — anchored on the [workspace] manifest above CARGO_MANIFEST_DIR; the target is 16/16 under nextest (#3126) Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(workspace-test): the quick tier's step cap 20 -> 60 min — measured killed at 20:00 under a 15/16-busy intel (run 34617807644) with every executed test green; the job's 150 remains the cap Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(tests): two dark aprender-core targets read repo-relative paths from the crate dir — they could not pass on any box The quick tier (#3089/#3115) ran realizar_integration_tests and toyota_principles_tests for the first time on this PR's workspace-test (run 34620804791, yoga-eph): integration_spec_complete panicked 'Spec file must exist (archived)' and nine toyota rows failed, because every path was written relative to the REPO root while cargo/nextest run tests with the CRATE dir (crates/aprender-core) as cwd. p13/p14 were vacuous for the same reason ('if spec_path.exists()' skipped the body). Repo files now resolve from CARGO_MANIFEST_DIR/../.. (the monorepo_invariants.rs pattern), crate sources (src/audio/stream.rs, src/format/v2.rs, src/audio/mel.rs) from CARGO_MANIFEST_DIR, and p8b judges unsafe_code on the workspace root manifest (the crate inherits [lints] workspace = true). Local: 26/26 + 47/47 under nextest. Pmat-Ticket: PMAT-1102 * fix(ci_resolve_dirty): --apply pushed nothing — it removed the worktree that held the merge Under --apply, resolve_one() merged origin/main into each DIRTY PR branch inside a throwaway worktree, printed a push hint, and then immediately ran git worktree remove -f. The merge commit became unreachable and nothing was pushed, forcing operators to redo merges by hand. This changes --apply to push the merge commit directly. We also add --no-push to preserve the old behaviour but keep the merge reachable by creating a local branch 'resolve/<pr>' before removing the worktree. Hermetic selftests are extended to prove these behaviours. Pmat-Ticket: PMAT-1098 * ci(actionlint): declare the runner labels the org actually carries (yoga, ephemeral, docker, build, ARM64, gb10, ada, perf-solo) Measured from gh api orgs/paiml/actions/runners 2026-09-11; the gpu-quick/cuda-unit jobs (67-C1/D1) ask for yoga/ephemeral/docker and actionlint flagged them unknown. Pmat-Ticket: PMAT-1098 * train A: tree-reader registry re-derived on the merged tree (+aprender-gpu driver::memory::transfer); two triage-receipt rows quoted issue titles carrying throughput literals — elided (claim-literal ratchet) Run 34634920736: workspace-test step 'Decide the test tier' exit 2 (registry drift: > aprender-gpu --lib driver::memory::transfer); guard-cargo step 'No claim literals on user-facing surfaces' exit 1 on docs/audits/triage-PMAT-3124-2026-09-11.md:76-77. Both are the train's own tree, not ENV. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(guard): check_tree_reader_tests.sh derived a box-dependent registry — `printf | grep -q` under pipefail fails on EPIPE when grep matches first, so an excluded crate was not skipped Run 34634920736 workspace-test step 'Decide the test tier': 'line 227: printf: write error: Broken pipe' then drift '> aprender-gpu --lib driver::memory::transfer' (derived only). Locally --update produced no change: the race depends on the box. wired_targets() skipped excluded crates with `printf '%s\n' "$ex" | grep -qxF "$c" && continue`; grep -q exits on its first match, printf gets EPIPE, pipefail makes the pipeline non-zero, `continue` never runs. Every producer|grep -q in the script is now a here-string (no producer process to kill): line 227 plus the self-test's row helper and fixture rows. The SIGPIPE+pipefail class (memory feedback_sigpipe_pipefail_false_red). --self-test 25 rows ok, --check 0, bashrs findings not grown. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(tests): falsify_iter7 match learns AttentionType::HybridGatedDeltaNet — the batch compiled a target no row's quick tier did Train run 34639757080 workspace-test (yoga-build2): error[E0004] non-exhaustive patterns at crates/aprender-core/tests/includes/falsify_iter7.rs:240 — #3099's 33c79fdb9 added the variant to AttentionType and its own quick tier never compiled falsification_model_oracle_tests (aprender-core untouched by that row); the train touches aprender-core, so the batch did. The arm asserts the GQA-shaped bound (kv_heads <= heads) on every size; the unsupported-on-CPU/GPU contract stays with the refusal tests. Local: 116/116. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(tests,ci): spec_checklist_19_inference read 13 repo paths from the crate dir (z10 dark on every box); guards-nightly derives the sccache host dir instead of shipping /home/<user> Train run 34641537515: workspace-test (yoga-build3) z10_zero_overhead_serving 'Specification should exist: NotFound' at spec_checklist_19_inference.rs:250 — every let-bound path in the file now anchors on workspace_root() (CARGO_MANIFEST_DIR/../..), so the 12 'if let Ok(content) = read_to_string(..)' rows stop being vacuous too; local: the whole target under nextest. guard-tree (intel-clean-room-5) step 43: the hardcoded-path ratchet refused guards-nightly.yml for the literal /home/noah/data/sccache the 67-E3 move copied from ci.yml (baselined there, new here); the nightly now derives SCCACHE_HOST_DIR=$HOME/data/sccache and falls back to $RUNNER_TEMP/sccache (cold) inside an ephemeral container. The class is filed as #3130 (77 more vacuous sites in 8 checklist files, sweep in flight). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(tests): eight spec_checklist targets read repo paths from the crate dir -- 77 vacuous rows now read the tree (#3130) Defect: integration tests under crates/aprender-core/tests/ bind repo-relative paths as bare string literals but cargo/nextest run a test binary with cwd = the CRATE dir, so the file is never found; most rows hid it behind if let Ok and were vacuous. Anchor: Added workspace_root() helper to resolve paths correctly from the crate directory to the repo root. Summary Lines: - spec_checklist_f_wasm: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s - spec_checklist_q_qwen_coder: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 118.50s - spec_checklist_r_model_import: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s - spec_checklist_t_realizar: ok. 25 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s - spec_checklist_u_performance: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s - spec_checklist_v_sovereign: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s - spec_checklist_w_advanced_perf: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s - spec_checklist_x_anti_stub: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(tests,ci): api_coverage asserted n=5 on a ONE-only ChoiceCount (dark row, impossible since the type landed); pr-review-receipt pinned to X64 until reject-76-drop dies on arm64 (#3132) Train run 34648948737: workspace-test (intel-6) test_chat_completion_request_n_multiple set ChoiceCount::ONE and asserted '"n":5' — the deserializer refuses n!=1 by design, so the row could never pass; it was dark because aprender-serve's integration targets run only when the crate is touched (#3130) and the train touches it (#3110/#3113/#3099). The row now asserts the contract both ways (ONE serialises as 1; n=5 is refused with the client-visible message); 269/270 locally, the one local-only red (test_completion_request_with_all_params, f32-narrowed temperature vs Some(0.7)) does not reproduce in CI's feature-unified build and is not touched here. pr-review-receipt (gx10-pool1): mutate-guard.sh 232/233 — reject-76-drop survived on arm64 exactly as memory recorded for gx10-pool2; the job goes back to X64 with the issue named in the comment. Not a required check (ruleset: gate, workspace-test). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(tests): api_coverage all-params row asserts the documented f32 narrowing of temperature — a round trip of 0.7 is f64::from(0.7f32) by contract Train run 34658698055: workspace-test (intel) test_completion_request_with_all_params: left Some(0.699999988079071) right Some(0.7). deserialize_temperature_f64 narrows through f32 on purpose (types.rs: 'the value still narrows to f32 before it reaches a sampler, so the narrowing is checked here too') and returns the narrowed value, so 0.7 can never round-trip exactly; the row was dark (#3130 class) and could never pass. It now asserts the contract. Local: api_coverage 270/270. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(tests): api_integration explain row asserts the endpoint's real refusal (501 Not Implemented) instead of a {200,400,503} set none of which is the contract Train run 34661764325: workspace-test test_apr_explain_exercises_handler failed on 'status in {OK, BAD_REQUEST, SERVICE_UNAVAILABLE}'. Since aprender#2375(2) /v1/explain refuses to fabricate attributions: with the demo APR model loaded (AppState::demo) a well-formed request is answered NOT_IMPLEMENTED (measured 501). The row was dark (#3130 class) and could never pass; it now asserts the exact outcome. Local: api_integration 32/32. Pmat-Ticket: PMAT-…

noahgift added this to the 0.67.0 milestone Sep 10, 2026

noahgift enabled auto-merge September 10, 2026 13:53

noahgift and others added 15 commits September 10, 2026 16:29

ci(release): the bullseye CUDA build left a root-owned target/ mountp…

d457931

…oint in the workspace — pre-create it as the runner (rebuild 34488955316 died at mkdir after a green 1m54s build) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge branch 'PMAT-1098-build-pool-any-of-three' into PMAT-1096-cuda-…

629869c

…asset-target-mountpoint

Merge commit '3b56d009bc256be68c4f1d17fca32795b0b16b9a' into PMAT-109…

b69e819

…8-67-F1-pp26-bytecompare-rest

ci(build-pool): the pr-review-* jobs stay on clean-room — a non-requi…

f5732c0

…red review job held yoga's only pool slot for 15 min while required jobs queued (#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

ci(build-pool): vendored-schemas on the any-arch pool — pure scripts,…

3e62921

… so gx10-build's idle cores may take it (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

ci(build-pool): the reusable sovereign-ci jobs run on the build pool …

8a4388a

…too (runs_on input, paiml/.github#67; operator: move now) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge commit '8a4388aeb7656c2421388344b6aa8d7996031c85' into PMAT-109…

d32dd23

…8-67-F1-pp26-bytecompare-rest

Merge remote-tracking branch 'origin/PMAT-1096-cuda-asset-target-moun…

1435689

…tpoint' into PMAT-1098-build-pool-any-of-three # Conflicts: # .github/workflows/ci.yml

Merge commit '6ee5b9b5628f26faee398eddc826e24383e62414' into PMAT-109…

07ad1d7

…8-67-F1-pp26-bytecompare-rest

ci: revert the ci.yml build-pool edits — routing lives in runner labe…

10c0f53

…ls (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Merge branch 'main' into PMAT-1098-67-F1-pp26-bytecompare-rest

89d66f0

noahgift mentioned this pull request Sep 10, 2026

fix(test): pp066 --v15-red fetches its v1.5 commit on a shallow checkout — every fresh yoga guard-tree died exit 128 #3108

Closed

Merge branch 'main' into PMAT-1098-67-F1-pp26-bytecompare-rest

36f040e

noahgift added this pull request to the merge queue Sep 11, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 11, 2026

noahgift mentioned this pull request Sep 12, 2026

PP-26 witness RED on gx10 (GB10 sm_121, CUDA 13.3): c=16 batch loses slot invariance at token 31; c=4 and lambda sm_89 stay invariant to 128 #3096

Open

Merge branch 'main' into PMAT-1098-67-F1-pp26-bytecompare-rest

8ef2449

noahgift added this pull request to the merge queue Sep 12, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 12, 2026

noahgift enabled auto-merge September 12, 2026 09:32

noahgift disabled auto-merge September 12, 2026 09:41

noahgift enabled auto-merge September 12, 2026 09:41

noahgift mentioned this pull request Sep 12, 2026

workspace-test on ARM64: gx10 runs the quick tier 3–4× faster than intel — route it arch-neutral once the four aarch64-only reds are fixed #3138

Closed

noahgift added this pull request to the merge queue Sep 12, 2026

Merged via the queue into main with commit a1645ce Sep 12, 2026
16 of 20 checks passed

noahgift deleted the PMAT-1098-67-F1-pp26-bytecompare-rest branch September 12, 2026 10:00

noahgift mentioned this pull request Sep 12, 2026

release: 0.67.0 #3145

Merged

Navigation Menu

Uh oh!

ci(cuda-nightly): 67-F1 — PP-26 byte-compare via REST with the job token; gx10 has no gh (#3096 filed for the witness itself) - #3097

ci(cuda-nightly): 67-F1 — PP-26 byte-compare via REST with the job token; gx10 has no gh (#3096 filed for the witness itself)#3097
noahgift merged 18 commits into
mainfrom
PMAT-1098-67-F1-pp26-bytecompare-rest

noahgift commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

noahgift commented Sep 11, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

noahgift commented Sep 10, 2026

67-F1 (PMAT-1098, epic #3078) — cuda-nightly's PP-26 byte-compare read the previous witness through gh, which no CUDA box has

Uh oh!

github-actions Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

noahgift commented Sep 11, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

67-F1 (PMAT-1098, epic #3078) — cuda-nightly's PP-26 byte-compare read the previous witness through `gh`, which no CUDA box has

github-actions Bot commented Sep 10, 2026 •

edited

Loading