ci(gpu): 67-C1/D1 — advisory gpu-quick (gx10) and cuda-unit (yoga) PR jobs that skip the hosts a change does not touch - #3095
Closed
noahgift wants to merge 42 commits into
Closed
ci(gpu): 67-C1/D1 — advisory gpu-quick (gx10) and cuda-unit (yoga) PR jobs that skip the hosts a change does not touch#3095noahgift wants to merge 42 commits into
noahgift wants to merge 42 commits into
Conversation
Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BjhtNUSensCYpQb3mCYLod
…ier does not do yet The quick tier turns 41 tree-reader targets into a 26-invocation `&&` chain (one cargo per crate, serial, 26 compiles of the shared deps): 55 min on a one-file YAML PR, timed out at 60 under fleet load (#3070). The replacement is ONE build graph filtered by a nextest filterset, so the translation from the registry tokens to that expression is the new load-bearing step. These rows fail now (--filterset is not a flag yet) and pin BOTH polarities: each recognised token becomes exactly one clause, and an unrecognised token is ENV (exit 2) rather than a silently dropped target — a dropped token is a tree-reader test that stops running while the step stays green, which is the failure mode the registry exists to prevent. Row 22 counts clauses against registry lines so nothing can be dropped in bulk either. Row 21 was written fail-open first (`grep -q ":--" && LEFTOVER || NONE-LEFT` passes on EMPTY output) and is hardened here before it was ever green. Refs #3084 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GREEN for the rows added in the previous commit.
`scripts/ci_test_tier.sh --filterset` turns the tree-reader registry's tokens
into a cargo-nextest filterset — `(package(c) & kind(lib))`, `(package(c) &
kind(bin))`, `binary_id(c::name)`, UNIONed with `|`. ci.yml's part-2 quick-tier
step now runs ONE `cargo nextest run --profile ci --workspace --lib --tests
--exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute
-E "$EXPR"` instead of a `&&` chain of one cargo per crate.
The binary-id forms are nextest's own and were VERIFIED on this workspace with
cargo-nextest 0.9.132 (`cargo nextest list --message-format json`), not assumed:
a lib suite's id is the bare package name, an integration target's is
`package::target`, a bin's is `package::bin/name`.
MEASURED on lambda, warm target dir (both listings from the same tree):
one graph 65263 tests, 41 binaries, 39 suites with >=1 test
build+list 96s, execution 220.2s, 316s wall, rc=0, 65263 passed
20-crate 65204 tests, 40 suites
chain (20 distinct crates, not 26 — the 41 tokens group into 20)
The two sets are NOT identical and the difference is entirely FEATURE
UNIFICATION, never target selection:
* A\B = 258 tests. Features the workspace resolve unifies ON and a
per-package resolve leaves OFF (aprender-core hf_hub / inspect::safetensors
/ format::homomorphic / format::quantize, aprender-orchestrate mcp_json,
aprender-test-lib async driver).
* B\A = 199 tests, and ZERO of them exist under the workspace resolve at all —
they are cfg(feature)-gated out. 197 are aprender-test-lib's browser mock
suites, 2 are aprender-present-cli::gate_can_fail. So the FULL tier, which
is `--workspace --lib`, never ran them either: the old chain was compiling a
DIFFERENT aprender-test-lib than the tier it is supposed to approximate.
Aligning the quick tier with the full tier's resolve is the fix, not a loss.
* Suite selection is otherwise exact: 41 clauses -> nextest reports "Starting
65263 tests across 41 binaries". The two suites with no tests are
aprender-core::setfit_conformance and ::falsification_spec_v10_tests, built
and selected in BOTH but empty under default features (they are the full
tier's --features setfit targets).
* `--bins` maps to `kind(bin)`, not to the whole package. The old chain ran
`cargo nextest run -p aprender-present-cli` with NO flags, so it also swept
up `gate_can_fail` — an integration target that is in neither the registry
nor the unwired ledger, i.e. not a tree reader at all. `kind(bin)` selects
exactly what the token claims: the cfg(test) unit tests inside the bins of a
crate that has no src/lib.rs (which is the only reason the token exists —
`--lib` there is `error: no library targets found`).
Both quick-tier steps drop to `timeout-minutes: 20`. 60 is the number that let
this step burn 55 minutes on a one-file YAML PR (run 34449608126) and then die
at the cap under fleet load on #3063 (#3070); the budget is the assertion.
check_tree_reader_tests.sh is untouched and still derives the same 41 targets:
its `full_tier_excludes` regex requires `--workspace --lib` followed IMMEDIATELY
by ` --exclude`, and the new line has `--tests` in between, so it still reads the
full tier's line and only that one (verified by running the regex).
Refs #3084
Closes #3070
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…RED at row 0 Ten rows, both polarities: a scratch PATH with every default tool is GREEN and the same PATH minus jq is RED naming MISSING jq; the JSON side-channel parses and records the missing tool; a usage error is exit 2 and a missing tool is exit 1; and the wiring itself is checked — every self-hosted job in fleet-toolset.yml, binary-release.yml and cuda-nightly.yml must run the preflight immediately after checkout, mutation-verified by deleting the step from a workflow copy. Run 34448908554 built a 21 MB CUDA asset on gx10 and died on the upload with `gh: command not found`. Nothing checked the toolset a workflow assumes. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…build scripts/ci_self_hosted_preflight.sh — defaults jq curl git python3 tar sha256sum rustup cargo; --cuda adds nvidia-smi AND asserts a device is actually visible; --need takes the extras a job knows it wants. `gh` is deliberately NOT a default: the CUDA lane moved to REST with curl+python3 because fleet boxes have no gh (#3074), and defaulting it would re-assert the assumption that failed. Exit 1 is a finding about the BOX, exit 2 a finding about the STEP that called this — merged codes send the wrong person. Identity (runner/labels/arch/glibc, driver under --cuda) is printed and, when PREFLIGHT_OUT or RUNNER_TEMP is set, written as JSON: that file is the fleet probe output. 13-row hermetic case table, both polarities, every tool a stub in a scratch PATH so the verdict does not depend on this box. Two rows were RED for the wrong reason first and are recorded as comments: the script called dirname/basename (now parameter expansion) and the nvidia-smi stub catted a fixture, so --cuda reported zero devices because `cat` was not on the scratch PATH. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…spends an hour
The preflight is now the step immediately after checkout in all seven
self-hosted jobs: binary-release.yml build-apr-cuda (--cuda --need docker
objdump), smoke-cuda (same; both GPU boxes carry `docker` as a runner LABEL, so
a pool that stopped matching its own labels is worth one second), and the two
clean-room jobs verify-cuda-assets and summary, which had no checkout at all;
plus the gx10 lane in cuda-nightly.yml. Not before `decide` and not before the
checkout there: on a yield night the box has no working tree, and a step that
bashes a repo path would exit 127 and turn a deliberately-yielded night RED.
fleet-toolset.yml probes all three pools daily at 04:47 UTC and uploads
toolset-<label>/preflight.json, 90-day retention, plus one table in the run
summary. ARTIFACT + SUMMARY, NOT A PR: a PR per day against evidence/ is ~365
machine-written PRs a year through a queue that runs at ~1 PR/hour, so the cost
of the record would exceed the record. A zero-row roll-up exits 1 — no
preflight.json downloaded is a broken probe, not a clean fleet.
The falsifier built its universe from the wrong side and said so: `runs-on`
alone saw 3 jobs, missing build-apr-cuda and smoke-cuda, whose selector is
`${{ fromJSON(matrix.labels) }}` — the two GPU jobs this row exists for. It now
reads the strategy block too, and sees 7.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…of paying an hour for a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s concurrently, deterministically and fail-closed Row (e) is the discriminator and it reads no clock: two planted guards rendezvous through marker files, so a serial dispatcher fails because the peer never started. Rows (a)-(d),(f) pin the universe, the named failure, the count, byte-identical ordering across two runs, and a guard that dies without a verdict counting as FAILED. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… judge in 20 Measured on run 34449608126 (PR #3074): guard-cargo 43.8 min over 80 steps, guard-tree 27.3 min over 45, and 18.4 of guard-tree's minutes were ONE step running ~35 independent guards one at a time on a 48-core box. That step now dispatches through an xargs pool (bounded -P 8, the clean-room pool is 16 runners on one host): 289 s -> 96 s locally on the real universe, same 41 checks. Seven tree-universe steps (26.8 min: the tier case table, model-tests, book examples, aprender-profile, the fd-0 scan, publish safety, wasm32) move to guards-nightly.yml with their name, run block and docker/mount shape verbatim, so check_guards_are_wired.sh -- which greps every workflow file, now pinned by rows 5-7 of its own case table -- still finds them wired. Both jobs take timeout-minutes: 20, the operator's budget, which is SMALLER than BSE-05's answer and overrides it. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dger, not a ratchet — the one line 67-E3's worker could not write (out of its scope) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nd two PR jobs no runner has ever seen Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d — 67-C1 gx10, 67-D1 yoga Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…7-C1-D1-gpu-pr-jobs
…ted) so a GPU-labelled job on it may carry concurrency perf-yoga — the one out-of-scope entry 67-C1/D1's worker could not write Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…yaml after declaring yoga (build.rs refuses drift, PMAT-958) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
noahgift
enabled auto-merge
September 10, 2026 13:24
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
…hosted-preflight # Conflicts: # .github/workflows/binary-release.yml
…7-C1-D1-gpu-pr-jobs
…oint in the workspace — pre-create it as the runner (rebuild 34488955316 died at mkdir after a green 1m54s build) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rst 'Host layout' step exports CI_TARGETS_ROOT / CI_CARGO_ROOT / SCCACHE_HOST_DIR / CI_REGISTRY / IMAGE with the intel clean-room defaults, 33 hardcoded sites read them; byte-identical on intel (#3100) Also: check_runner_labels.sh accepts the `build` pool label. No job changes where it runs yet (BP-3). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…asset-target-mountpoint
… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…8-67-E3-guards-under-budget # Conflicts: # .github/workflows/ci.yml
…C1-D1-gpu-pr-jobs
… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…red review job held yoga's only pool slot for 15 min while required jobs queued (#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… so gx10-build's idle cores may take it (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…too (runs_on input, paiml/.github#67; operator: move now) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…8-67-E3-guards-under-budget
…8-67-C1-D1-gpu-pr-jobs
…tpoint' into PMAT-1098-build-pool-any-of-three # Conflicts: # .github/workflows/ci.yml
…pool; guard-tree/guard-cargo/mutants/vendored-schemas/pr-review-* and the reusable ci jobs back on clean-room until measured (ci / security refuses to run without pmat, absent on yoga; #3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…8-67-E3-guards-under-budget # Conflicts: # .github/workflows/ci.yml
…8-67-C1-D1-gpu-pr-jobs # Conflicts: # .github/workflows/ci.yml
…t pool-free ci.yml ⊕ main) — keeps this branch's own guard changes (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nd give guard-cargo's remaining cargo steps the job's own CARGO_HOME — moving model-tests to the nightly had left guard-cargo with none, and guard_tree_test's case table requires it Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nd give guard-cargo's remaining cargo steps the job's own CARGO_HOME — moving model-tests to the nightly had left guard-cargo with none, and guard_tree_test's case table requires it Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y device to the RTX 4090's
test_context_memory_info and test_context_total_memory asserted `total > 20 GB` ("RTX 4090 should have
>20GB VRAM") on whatever device 0 is. yoga's RTX 4060 Laptop GPU has 8 GB, so this PR's cuda-unit job
on yoga-eph failed both while saying nothing true about the card. A per-device floor table keyed by the
CUDA device name now holds the 4090 to >20 GB (unchanged) and the RTX 4060 to >7 GB; an unlisted device
(gx10's GB10 reports unified memory) gets the structural checks only.
Proven on lambda's RTX 4090: both tests green, and a mutant raising the 4090 floor to 30 GB turns them RED
(exit 101). cargo fmt --check clean; cargo deny and the aprender-contracts lib tests green.
`cargo clippy -p aprender-gpu --features cuda --lib --tests -D warnings` reports 599 errors, none in this
change: 551 are the banned unwrap() across the cuda test files, and the two in this file are the
pre-existing line 13 (`assert!(... || true)`, overly_complex_bool_expr). CI does not lint these targets.
The cuBLAS half of this job's failure ("cuBLAS library not found") was the runner containers' loader
path, fixed on yoga and gx10 by LD_LIBRARY_PATH=/usr/local/cuda/lib64 (paiml/infra 4c2afe8).
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…er script from main ci.yml: workspace-test keeps main's measured 45-min timeout; sccache path from main; guard-tree keeps BOTH case-table steps (67-E3 dispatcher table + fleet hygiene tables); the BSE-03 SATD ratchet STAYS in guard-cargo (its universe is the diff vs the comparand; nightly HEAD==origin/main makes it vacuous there) and only 'Book rust examples compile' moves; guard-tree/guard-cargo job timeouts 20->30 (fleet-ledger p90 25/45 minus the moved ~23 min, 1.3x margin). scripts/ci_test_tier.sh: taken from main — the branch's copy predates PMAT-3119/3120 and adds nothing E3-specific (12 re-adds of older lines, 229 removals). Pmat-Ticket: PMAT-1098
… take the measured 30 Pmat-Ticket: PMAT-1098
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
67-C1 / 67-D1 — advisory GPU PR jobs that skip the hosts they do not need (PMAT-1098, epic #3078)
Stacked on #3088 (C2), #3089 (E2), #3094 (E3), #3070/E1 — the branch carries them so
ci / gatesees the whole graph; after they merge the diff narrows to the four commits below.What lands
scripts/ci_gpu_touched.sh(+scripts/tests/ci_gpu_touched_test.sh, 20 rows): derives fromHEAD^1..HEAD(or the PR range) which GPU hosts a change actually touches — CUDA source / kernels /aprender-gpu/aprender-cuda-edge→gx10; unit-level cuda tests →yoga; nothing GPU-relevant → both SKIP. Empty selection exits 0 withgpu_hosts=[], never a guess..github/workflows/ci.yml: three new jobs —gpu-touched(the selector, hosted on the clean-room pool, neverubuntu-*),gpu-quick([self-hosted, Linux, ARM64, cuda, gx10],concurrency: perf-gx10) andcuda-unit([self-hosted, Linux, X64, cuda, yoga],concurrency: perf-yoga). Both GPU jobs are advisory: not ingate.needs,continue-on-error: falseso a red is visible, but they cannot block a merge until 67-D0 (runner-group scoping) lands.scripts/perf-matrix.yaml+ re-vendoredcrates/aprender-test-lib/perf-matrix.vendored.yaml:hosts.yoga(RTX 4060 Laptop, sm_89,compute_class: cuda,ci_runner: self-hosted) —check_perf_concurrency_groups.shrequires every GPU job'sperf-<host>group to name a host in the matrix andaprender-test-lib/build.rsrefuses a vendored copy that drifts (PMAT-958). This is the one entry the phase worker could not write in-scope; the orchestrator wrote it and re-vendored.Acceptance (re-run by the orchestrator on cdd9fa1, 2026-09-10T13:12–13:17Z)
RED commit adc2378 pinned the 20 rows against a selector that did not exist and two jobs no runner had seen; 54f0984 turns them green.
Receipt:
docs/audits/impl-PMAT-1098-receipt.mdstaysstatus: partialper G-11 (a row PR never writes DAG/spec/roadmap counts); the orchestrator docs commit flips it. Hosted runners: none added (check_runner_labels.shgreen).Pmat-Ticket: PMAT-1098
🤖 Generated with Claude Code