fix(R-3): the cuBLAS-backward training banner is an event (launch counter), and --gpu-backend cuda on a cpu-only build is FeatureDisabled exit 9 — never a silent CPU run (PMAT-991, #2906) by noahgift · Pull Request #3001 · paiml/aprender

noahgift · 2026-09-06T03:44:42Z

PP-066 DAG row R-3 (spec §5 T-6, D-7; epic #2873; ticket PMAT-991; receipt docs/audits/impl-PMAT-991-receipt.md, status: partial until merged). feedback_apr_finetune_lora_cpu_only: the "cuBLAS backward" training banner was printed from the request, before any step, on any build; --gpu-backend cuda on a cpu-only build fell through to the CPU silently. Now the banner is an event and the impossible request is a refusal.

What lands

entrenar (crates/aprender-train): backward_kernel_launches() / reset_backward_kernel_launches() — an AtomicU64 bumped at every cuBLAS backward launch (cuda_forward/matmul.rs a, a_accumulate, b; matmul_f16.rs a, b); training_backend_banner(requested) -> Option<String> is Some iff requested == "cuda" and the counter > 0. RED-first test tests/banner_truth.rs (both feature arms; the cuda arm drives one cuBLAS backward on this box's RTX 4090 and watches the counter move).
apr finetune: gpu_backend_decision(requested, build_has_cuda, build_has_wgpu) — cuda/wgpu on a build without the feature is CliError::FeatureDisabled (exit code 9, read from error.rs:106); auto/cpu never refuse; pre_training_notice never says "cuBLAS backward"; post_training_banner prints entrenar's line after training. Case table commands/finetune_gpu_backend_truth_tests.rs (5 tests, both polarities).
contracts/apr-train-banner-truth-v1.yaml (kind: pattern, TBT-OB-001..003; pv validate valid, pv lint 0/0). README contract count 1811 → 1812.
test_run_training_creates_adapter hard-coded "cuda" and asserted the old silent fallback; it now names the CPU path it always ran.

Acceptance, re-run by the orchestrator on eaea476

A_i	rc
`cargo test -p aprender-train --test banner_truth`	0 (2 passed)
`cargo test -p aprender-train --test banner_truth --features cuda` (lambda, sm_89)	0 (2 passed)
`cargo test -p apr-cli --lib gpu_backend_truth`	0 (5 passed)
`cargo test -p apr-cli --lib`	0 (7213 passed, 12 ignored)
`cargo test -p aprender-train --lib`	101: 7621 passed, 3 failed in `prune::snapshot_tests` — identical on a pristine origin/main worktree at `65680cd` while CI's workspace-test there is green: local insta drift, `src/prune` untouched here
`cargo fmt --all -- --check` · `cargo clippy -p {aprender-train,apr-cli} --lib -- -D warnings`	0 · 0 · 0
`check_contract_test_binding.sh` · `check_contract_enforcement.sh` · `check_readme_claims.sh` · `check_no_claim_literals.sh` · `check_roadmap_diff_additive.sh`	0 ×5

Mutations — RED, then restored GREEN

mutation	result
`training_backend_banner` returns Some for "cuda" regardless of the counter	`banner_is_none_for_cuda_with_zero_launches` FAILED, `cpu_only::cpu_backward_never_increments_the_device_counter` FAILED (0 passed, 2 failed) → restored 2 passed
`gpu_backend_decision` returns `Ok(Cpu)` for "cuda" on a build without cuda	`cuda_request_on_cpu_only_build_is_a_refusal_not_a_fallback` FAILED, `gpu_backend_decision_case_table` FAILED (3 passed, 2 failed) → restored 5 passed

Findings recorded in the receipt

The crate dir is aprender-train, the lib is entrenar (S0-10 class).
crates/apr-cli/src/lib.rs cannot be committed on this box: the pre-commit complexity hook follows its include!() graph and refuses on pre-existing debt (dispatch.rs:103 cognitive 41, dispatch.rs:508 cognitive 30, help_producer_truth.rs:51 cognitive 73). The integration-test seam was dropped; the case table lives in commands::finetune's own test module. Not discharged with #[allow] or --no-verify.
Not instrumented: custom-PTX backward kernels under autograd/cuda_backward/ — a PTX-only backward would leave the banner None (open question for the 3-lane review; the spec names the cuBLAS sites).
gx10 leg not run here (lambda only); C4 dogfood covers it.

Review round (3-lane quorum, 2026-09-06 — 3/3 mergeable with changes; applied at a137ba6, re-verified)

Unanimous: the process-wide counter with a test-only reset leaked across fine-tunes → the banner is now scoped to the run (training_backend_banner(requested, launches_at_start), snapshot before trainer.train(), no reset API); the tests snapshot and compare (race-free under parallel tests).
Lane 1 alone said gemm_f16_to_f32_backward_a launched uncounted; lanes 2/3 said all sites were covered. Lane 1 was right, and three more uncounted device backward launches were found in matmul.rs (gemm_nf4_backward_a_cublas, the NF4 PTX and tensor-core backward_a — the QLoRA path). Nine launch sites are counted now.
-m lora --gpu-backend cuda now REFUSES (ValidationFailed, exit 5) instead of running the CPU path under a GPU flag; auto + plain LoRA is the CPU path and the notice says so (gpu_backend_decision gained method_has_cuda_path).
Dead gpu_backend_notice/GpuBackendPlan (still holding the request-derived cuBLAS string) and their five tests removed.
Re-run on a137ba6: banner_truth 2/2 (default and --features cuda), gpu_backend_truth 8/8, cargo test -p apr-cli --lib 7211 passed, cargo test -p aprender-train --lib 7621 passed + the same three pre-existing prune snapshot failures, clippy/fmt clean, pv validate valid. Mutations: 1 (banner unconditional) 0/2 passed; 2 (cuda on a cpu-only build → Cpu) 6/8; 3 (plain LoRA + cuda → Cuda, no refusal) 7/8; all restored GREEN.

Quorum verdicts and the adjudication are in the receipt. Receipt for this PR itself: advisory, not produced (driver A1).

no-close: #2873 is the epic this row implements one part of; an epic is not closed by one of its rows.

github-actions · 2026-09-06T03:45:22Z

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3001 head=4be654a38c8d28c760e06d398740c7fbc7f13557 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

…ived from launched backward kernels, not from the request Pmat-Ticket: PMAT-991

…launch; training_backend_banner derived from it (PMAT-991, #2906) Pmat-Ticket: PMAT-991

…ent, the cuda request on a cpu-only build is a refusal (exit 9) Pmat-Ticket: PMAT-991

…(exit 9); the cuBLAS backward line is printed after training from entrenar's launch counter, never from the request (PMAT-991, #2906) Pmat-Ticket: PMAT-991

…ne's test module (the lib.rs test seam is blocked by pre-existing complexity debt in lib.rs's include graph: dispatch.rs, help_producer_truth.rs); test_run_training_creates_adapter exercises the CPU path it always ran, and says so — 'cuda' on a cpu-only build is now a refusal (PMAT-991) Pmat-Ticket: PMAT-991

… banner is derived from launches, a gpu request the build cannot honour is FeatureDisabled (PMAT-991, #2906) README.md's contract count was already at 1812 on the rebased base (bumped by the G-11 ratchet process since this branch's original base); no README edit needed or made here (row PRs do not write README.md, driver rule). Pmat-Ticket: PMAT-991

… rows Pmat-Ticket: PMAT-991

…unch snapshot, no reset API; a second fine-tune in one process inherits nothing; race-free tests), four more device-side backward launch sites counted (f16->f32 a, NF4 cuBLAS/PTX/tensor-core a), -m lora --gpu-backend cuda is ValidationFailed exit 5 (refuse or train on the GPU), dead gpu_backend_notice removed (PMAT-991, #2906) Pmat-Ticket: PMAT-991

…able-with-changes, lane 1 right on the f16 site, three more NF4 sites found), mutations re-run at a137ba6 Pmat-Ticket: PMAT-991

noahgift · 2026-09-06T19:12:27Z

Rebased onto origin/main @ 3792afa3d (after G-10/G-11 #3011/#3020 landed). Dropped commit d5fde6b5b (chore(roadmap): PMAT-991 kind:code label) — it wrote docs/roadmaps/roadmap.yaml, which G-11a now forbids in a row PR. The README.md count-line bump in ccda35736 needed no action: the rebased base already carried 1812 (bumped since by the G-11 ratchet process), so the contract commit now adds only contracts/apr-train-banner-truth-v1.yaml with no README diff — row PRs do not write README.md per driver rule. The other 8 commits cherry-picked cleanly onto the new base with no conflicts.

Re-verified at HEAD (b5b2b044f):

cargo check -p apr-cli -p aprender-train --lib → clean
cargo test -p aprender-train --test banner_truth → 2/2 pass
cargo test -p apr-cli --lib finetune → 79/79 pass
cargo test -p apr-cli --lib gpu_backend → 10/10 pass (gpu_backend_truth_tests::*, test_finetune_rejects_unknown_gpu_backend_2583, test_finetune_accepts_its_own_gpu_backends_2583)
pv validate contracts/apr-train-banner-truth-v1.yaml → 0 errors, 0 warnings
bash scripts/check_row_pr_write_set.sh --base origin/main --head HEAD --branch agent/R-3 → PASS, 11 changed paths, no shared file touched

noahgift · 2026-09-06T21:48:08Z

Auto-merge armed. Currently BLOCKED in the queue on two unrelated fleet issues: #3028/PR #3030 (roadmap.yaml duplicate ids redding ci / security on every PR) and a fleet-wide gap in scripts/pmat_bin.sh's pinned analyser on some runner hosts (paiml/infra#468). Not this PR's diff — will merge once both clear.

…dogfood `pmat work add` was not used, for the two reasons this ticket exists: 1. It allocates from the roadmap it can see. Max across origin/main and every open pp-066 head is PMAT-1079, and three instances are minting concurrently, so an auto-mint races. 1090 leaves 1080-1089 as headroom. A gap is opaque; a collision is data loss (and 12 duplicate ids on stale heads is what turned `ci / security` red on #3031 and #3001). 2. It re-serialises the whole file — the defect this ticket fixes. The new `--staged` mode judged its own commit, which is the point: $ bash scripts/check_roadmap_diff_additive.sh --staged === roadmap.yaml STAGED diff is additive: base=c04eda87d (merge-base(origin/main, HEAD)) head=8e4ceee37 (index tree) === roadmap-diff: base=818 head=819 added=1 lifecycle=0 reserialised=0 deleted=0 PASS $ git diff --cached --stat docs/roadmaps/roadmap.yaml | 19 +++++++++++++++++++ added=1, reserialised=0, 0 deletions. That is the diff the rule asks for, and `pmat work add` cannot produce it — which is why the remedy script exists and why nothing calling it made the rule unenforceable in practice. Refs #3047, PMAT-1090. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUX4ymyt8XhjXbtTuNH8L

check_roadmap_diff_additive.sh reported base id(s) missing at head: this branch forked before they landed on main, and a plain update takes one side of the file whole. Resolved as main's roadmap plus this branch's own new entries, each at its sorted slot among same-prefix peers. The other three roadmap guards pass on the result. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…is the answer One conflict, docs/audits/impl-estimates.jsonl. Both sides APPENDED: this branch added four PMAT-991 rows, main added three PMAT-972 rows (via #3006). Taking either side whole deletes measurements nothing else records. Resolved as the union, main's rows first since they are already in main's history. Verified after: 46 rows, every one valid JSON, zero duplicates, no conflict markers. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…is the answer One conflict, docs/audits/impl-estimates.jsonl. Both sides APPENDED different tickets' rows; taking either side whole deletes measurements nothing else records. Resolved as the union, main's rows first since they are already in main's history — the same resolution #3001 needed an hour ago, for the same reason: a squash-merge from the queue makes every branch carrying this file DIRTY on it. Verified after: every row valid JSON, zero duplicates, no conflict markers. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

One conflict, docs/audits/impl-estimates.jsonl, append-only, resolved as the union with main's rows first. #3001 at ~07:05Z, #3093 at ~08:35Z, this at ~08:45Z — three identical resolutions in under two hours. #3256 is the fix: `docs/audits/*.jsonl merge=union` in .gitattributes, with a guard holding the scope (149 goldens and datasets must NOT be union-merged). Once that lands this conflict stops existing. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… the conflict was right Two conflicts. docs/audits/impl-estimates.jsonl — append-only, resolved as the union with main's rows first. Fourth time today (#3001, #3093, #3005, this); #3256 makes it stop. README.md — the CONTRACT_COUNT block, 1839 on this branch against 1837 on main. Taking either side would have been wrong: the branch ADDS contracts, so the merged tree is neither number. CLAUDE.md's rule for this table is explicit — "the table gives the command, and the value is only a dated sample. If a number here disagrees with its command, the command wins." find contracts -name '*.yaml' | wc -l -> 1840 Set to 1840 at both blocks, and check_readme_claims.sh agrees: PASS FALSIFY-README-002 contract_count: 1840 (CONTRACT_COUNT block, derived by scripts/readme_sync.sh; 1 block(s) agree with the merge tree 259c8f7) A generated value is not a text conflict to pick a side of. It is a question with a command attached. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…the union Both sides only APPEND to docs/audits/impl-estimates.jsonl: main added 3 PMAT-989 rows, this branch added 4 PMAT-991 rows, zero deletions on either side. The textual conflict has no semantic content — the resolution is all 7 rows, and every surviving row was re-parsed as JSON to prove it. It is the only file that changed on both sides (42 base rows -> 49). Pmat-Ticket: PMAT-991

…d the ratchet measures the MERGED tree FALSIFY-README-002 compares the CONTRACT_COUNT block against the merge tree, not against the diff: base=1840 merge=1841 delta=+1, because this branch adds contracts/apr-train-banner-truth-v1.yaml. The block said 1840. Regenerated with `make readme-sync`. check_readme_claims.sh 5/5 PASS. Pmat-Ticket: PMAT-991

…dy wrote (paiml#3256) docs/audits/impl-estimates.jsonl is appended one row per (ticket, phase). Two branches that both append conflict on the last line, so a squash-merge from the queue makes EVERY branch carrying it DIRTY the moment anything merges. Measured 2026-09-14: paiml#3001 at ~07:05Z and paiml#3093 at ~08:35Z — ninety minutes apart, same file, same conflict, and I resolved both identically (take the union, main's rows first, verify every row is valid JSON and no duplicates). Doing that by hand a third time would be the tell I had already missed twice. `union` is git's built-in driver. Proved in a throwaway repo BEFORE writing it here, because an attribute that is declared and does not resolve conflicts is exactly the theater this repo names most often: without .gitattributes a two-sided append leaves 1 conflict marker with merge=union 0 markers, result = base + MAIN + SIDE THE SCOPE IS THE POINT, AND IT IS WHY THIS SHIPS WITH A GUARD RATHER THAN A COMMENT. `*.jsonl merge=union` would be a defect, not a convenience: 149 of the 151 .jsonl files here are GOLDENS and DATASETS — crates/aprender-contrastive-data/tests/goldens/*, datasets/*, evidence/pr-review/*/receipt.intoto.jsonl. A union merge there silently DUPLICATES rows, destroying the byte-identity those files exist to assert. check_append_only_ledgers.sh holds that distinction as a case table, both polarities, and R3 proves the MECHANISM rather than the declaration: R1 the ledger resolves merge=union ok R2 a golden does NOT ok R3a without the attribute a two-sided append CONFLICTS (1 marker) ok R3b with merge=union it does not (0 markers) ok R3c and the result is the UNION, not one side (3 lines) ok Mutation, the one that matters: widening .gitattributes to `*.jsonl merge=union` turns the guard RED with 149 findings — every golden and dataset named. The narrow scope is load-bearing and the guard says so. Measured on this tree: 2 append-only ledgers (impl-estimates.jsonl, impl-routing.jsonl — the latter written by route.sh record-event), 149 other .jsonl correctly excluded. bashrs 0 errors; check_guards_are_wired PASS, unwired baseline did not grow. NOT VERIFIED, and I am not claiming it: whether GitHub's SERVER-SIDE mergeability computation honours `.gitattributes merge=union`. If it does, these PRs stop going DIRTY at all; if it does not, the local resolution becomes a no-op instead of a hand-edit. Either way this is an improvement, but only the local half is proved above. Pmat-Ticket: PMAT-1098 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

noahgift · 2026-09-20T14:02:39Z

stale — APR-RELEASE-001 §6, applied mechanically 2026-09-20.

No activity since 2026-09-17 13:39Z; 2 release trains have been cut since (v0.68.2, v0.68.1). Age is the only input to this sweep — it is not a judgement on the work, and a train label such as pp-066/carry-* is not an exemption.

What happens next: if one more train passes while this is still labelled stale, §6 closes it. The branch is kept — closing a PR here discards a review thread, never commits.

To clear it: push, rebase, or say on the thread what it is waiting for. Any of the three removes the label at the next sweep. If it is blocked on something external, name that here — a blocker with an owner is not sprawl, and it stops the clock.

noahgift · 2026-09-23T21:54:48Z

closed by 0.70 batch triage: no quorum receipt and no live owner; branch kept, reopen with an owner

Authority: per operator order (quoted) "yes, clearn up. no sovereign repo can have triage, ticket, branche issues and arbiter shoudl be doing this! tell that agent to get to work" + cop (aprender-cf) ruling on the close criteria (a draft, or >7d old with no live owner, and no quorum receipt). The operator did not select this PR individually.

noahgift enabled auto-merge September 6, 2026 03:44

noahgift added this pull request to the merge queue Sep 6, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Sep 6, 2026

This was referenced Sep 6, 2026

Epic: PP-066 — 0.66 performance-parity release #2873

Closed

feat(G-11): shared-file write contention — row PRs never write the DAG, roadmap, spec block or README counts; DAG status derived from receipts; README counts a ratchet (PMAT-1062) #3020

Merged

noahgift added 9 commits September 6, 2026 21:08

test(R-3): banner_truth — RED: the cuBLAS-backward banner must be der…

ec48284

…ived from launched backward kernels, not from the request Pmat-Ticket: PMAT-991

feat(R-3): backward_kernel_launches counter at every cuBLAS backward …

c950926

…launch; training_backend_banner derived from it (PMAT-991, #2906) Pmat-Ticket: PMAT-991

test(R-3): finetune_gpu_backend_truth — RED: the cuBLAS line is an ev…

b2e6af4

…ent, the cuda request on a cpu-only build is a refusal (exit 9) Pmat-Ticket: PMAT-991

feat(R-3): --gpu-backend cuda on a cpu-only build is FeatureDisabled …

6908bed

…(exit 9); the cuBLAS backward line is printed after training from entrenar's launch counter, never from the request (PMAT-991, #2906) Pmat-Ticket: PMAT-991

docs(audits): impl-PMAT-991 receipt (R-3, status: partial) + estimate…

9234c4e

… rows Pmat-Ticket: PMAT-991

docs(audits): impl-PMAT-991 receipt — review quorum record (3/3 merge…

b5b2b04

…able-with-changes, lane 1 right on the f16 site, three more NF4 sites found), mutations re-run at a137ba6 Pmat-Ticket: PMAT-991

noahgift force-pushed the agent/R-3 branch from 80bd086 to b5b2b04 Compare September 6, 2026 19:12

noahgift added inst:C PP-066 instance claim (I14): inst:C R-3 PP-066 DAG row R-3 pp-066 PP-066 (0.66) DAG row labels Sep 6, 2026

noahgift enabled auto-merge September 6, 2026 21:30

noahgift disabled auto-merge September 7, 2026 11:54

noahgift added this to the 0.68.0 milestone Sep 10, 2026

noahgift added the needs-rebase mergeStateStatus DIRTY: conflicts with main; rebase before it can enter the queue label Sep 11, 2026

noahgift mentioned this pull request Sep 12, 2026

Every open PR is roadmap-non-additive, so nothing can enter the merge queue and all three boxes sit near 10% — 31 of 31 measured #3195

Closed

noahgift enabled auto-merge September 13, 2026 06:24

github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Sep 14, 2026

noahgift enabled auto-merge September 14, 2026 07:09

noahgift mentioned this pull request Sep 14, 2026

fix(git): a resolution that is always the same is a merge driver nobody wrote #3256

Merged

Merge branch 'main' into agent/R-3

3eaee3b

noahgift disabled auto-merge September 14, 2026 17:42

noahgift enabled auto-merge September 14, 2026 18:37

Merge branch 'main' into agent/R-3

d024b73

noahgift disabled auto-merge September 14, 2026 22:42

noahgift added 3 commits September 15, 2026 07:58

Merge branch 'main' into agent/R-3

aa016d1

noahgift added 2 commits September 15, 2026 13:37

Merge branch 'main' into agent/R-3

fa24dfb

Merge branch 'main' into agent/R-3

4be654a

noahgift modified the milestones: 0.68.0, 0.70.0, 0.71.0 Sep 16, 2026

noahgift mentioned this pull request Sep 17, 2026

EPIC: PP-QUANT-001 — quant-type dispatch, Phase 0/1 (0.69 train) #3421

Closed

2 tasks

noahgift added the stale APR-RELEASE-001 §6: >=2 release trains with no activity label Sep 20, 2026

noahgift closed this Sep 23, 2026

noahgift deleted the agent/R-3 branch September 24, 2026 07:21

Navigation Menu

fix(R-3): the cuBLAS-backward training banner is an event (launch counter), and --gpu-backend cuda on a cpu-only build is FeatureDisabled exit 9 — never a silent CPU run (PMAT-991, #2906) - #3001

fix(R-3): the cuBLAS-backward training banner is an event (launch counter), and --gpu-backend cuda on a cpu-only build is FeatureDisabled exit 9 — never a silent CPU run (PMAT-991, #2906)#3001
noahgift wants to merge 26 commits into
mainfrom
agent/R-3

noahgift commented Sep 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

noahgift commented Sep 6, 2026

Uh oh!

noahgift commented Sep 6, 2026

Uh oh!

noahgift commented Sep 20, 2026

Uh oh!

noahgift commented Sep 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

noahgift commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

noahgift commented Sep 6, 2026

Uh oh!

noahgift commented Sep 6, 2026

Uh oh!

noahgift commented Sep 20, 2026

Uh oh!

noahgift commented Sep 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

noahgift commented Sep 6, 2026 •

edited

Loading

github-actions Bot commented Sep 6, 2026 •

edited

Loading