fix(compute,aarch64): gx10 could not run ci / lint — ARM-only clippy errors, and #2567's parallel Q4_K path was never called (PMAT-1102) - #3112
Open
noahgift wants to merge 43 commits into
Open
Conversation
Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BjhtNUSensCYpQb3mCYLod
…ier does not do yet The quick tier turns 41 tree-reader targets into a 26-invocation `&&` chain (one cargo per crate, serial, 26 compiles of the shared deps): 55 min on a one-file YAML PR, timed out at 60 under fleet load (#3070). The replacement is ONE build graph filtered by a nextest filterset, so the translation from the registry tokens to that expression is the new load-bearing step. These rows fail now (--filterset is not a flag yet) and pin BOTH polarities: each recognised token becomes exactly one clause, and an unrecognised token is ENV (exit 2) rather than a silently dropped target — a dropped token is a tree-reader test that stops running while the step stays green, which is the failure mode the registry exists to prevent. Row 22 counts clauses against registry lines so nothing can be dropped in bulk either. Row 21 was written fail-open first (`grep -q ":--" && LEFTOVER || NONE-LEFT` passes on EMPTY output) and is hardened here before it was ever green. Refs #3084 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GREEN for the rows added in the previous commit.
`scripts/ci_test_tier.sh --filterset` turns the tree-reader registry's tokens
into a cargo-nextest filterset — `(package(c) & kind(lib))`, `(package(c) &
kind(bin))`, `binary_id(c::name)`, UNIONed with `|`. ci.yml's part-2 quick-tier
step now runs ONE `cargo nextest run --profile ci --workspace --lib --tests
--exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute
-E "$EXPR"` instead of a `&&` chain of one cargo per crate.
The binary-id forms are nextest's own and were VERIFIED on this workspace with
cargo-nextest 0.9.132 (`cargo nextest list --message-format json`), not assumed:
a lib suite's id is the bare package name, an integration target's is
`package::target`, a bin's is `package::bin/name`.
MEASURED on lambda, warm target dir (both listings from the same tree):
one graph 65263 tests, 41 binaries, 39 suites with >=1 test
build+list 96s, execution 220.2s, 316s wall, rc=0, 65263 passed
20-crate 65204 tests, 40 suites
chain (20 distinct crates, not 26 — the 41 tokens group into 20)
The two sets are NOT identical and the difference is entirely FEATURE
UNIFICATION, never target selection:
* A\B = 258 tests. Features the workspace resolve unifies ON and a
per-package resolve leaves OFF (aprender-core hf_hub / inspect::safetensors
/ format::homomorphic / format::quantize, aprender-orchestrate mcp_json,
aprender-test-lib async driver).
* B\A = 199 tests, and ZERO of them exist under the workspace resolve at all —
they are cfg(feature)-gated out. 197 are aprender-test-lib's browser mock
suites, 2 are aprender-present-cli::gate_can_fail. So the FULL tier, which
is `--workspace --lib`, never ran them either: the old chain was compiling a
DIFFERENT aprender-test-lib than the tier it is supposed to approximate.
Aligning the quick tier with the full tier's resolve is the fix, not a loss.
* Suite selection is otherwise exact: 41 clauses -> nextest reports "Starting
65263 tests across 41 binaries". The two suites with no tests are
aprender-core::setfit_conformance and ::falsification_spec_v10_tests, built
and selected in BOTH but empty under default features (they are the full
tier's --features setfit targets).
* `--bins` maps to `kind(bin)`, not to the whole package. The old chain ran
`cargo nextest run -p aprender-present-cli` with NO flags, so it also swept
up `gate_can_fail` — an integration target that is in neither the registry
nor the unwired ledger, i.e. not a tree reader at all. `kind(bin)` selects
exactly what the token claims: the cfg(test) unit tests inside the bins of a
crate that has no src/lib.rs (which is the only reason the token exists —
`--lib` there is `error: no library targets found`).
Both quick-tier steps drop to `timeout-minutes: 20`. 60 is the number that let
this step burn 55 minutes on a one-file YAML PR (run 34449608126) and then die
at the cap under fleet load on #3063 (#3070); the budget is the assertion.
check_tree_reader_tests.sh is untouched and still derives the same 41 targets:
its `full_tier_excludes` regex requires `--workspace --lib` followed IMMEDIATELY
by ` --exclude`, and the new line has `--tests` in between, so it still reads the
full tier's line and only that one (verified by running the regex).
Refs #3084
Closes #3070
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…of paying an hour for a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r a moved main Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…oint in the workspace — pre-create it as the runner (rebuild 34488955316 died at mkdir after a green 1m54s build) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rst 'Host layout' step exports CI_TARGETS_ROOT / CI_CARGO_ROOT / SCCACHE_HOST_DIR / CI_REGISTRY / IMAGE with the intel clean-room defaults, 33 hardcoded sites read them; byte-identical on intel (#3100) Also: check_runner_labels.sh accepts the `build` pool label. No job changes where it runs yet (BP-3). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…asset-target-mountpoint
… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…8-67-E2-queue-mirrors-pr
… box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; #3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…red review job held yoga's only pool slot for 15 min while required jobs queued (#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… so gx10-build's idle cores may take it (#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…too (runs_on input, paiml/.github#67; operator: move now) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…8-67-E2-queue-mirrors-pr
…tpoint' into PMAT-1098-build-pool-any-of-three # Conflicts: # .github/workflows/ci.yml
…pool; guard-tree/guard-cargo/mutants/vendored-schemas/pr-review-* and the reusable ci jobs back on clean-room until measured (ci / security refuses to run without pmat, absent on yoga; #3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…8-67-E2-queue-mirrors-pr
…ls (#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ustc-sccache wrapper runs under dash, which drops env vars whose names are not shell identifiers, so CARGO_BIN_EXE_aprender-explain never reached rustc
The quick tier (one workspace build over --lib --tests) is the first CI step to compile aprender-explain's integration test, whose binary name has a hyphen: env!("CARGO_BIN_EXE_aprender-explain") failed 'not defined at compile time' on intel and yoga. Proven: through /bin/sh (dash) CARGO_BIN_EXE_aprender-explain is dropped and CARGO_BIN_EXE_apr survives, in the sovereign-ci image and locally; the step's exact build WITHOUT the wrapper compiled clean on yoga (12m42s). All 8 RUSTC_WRAPPER settings now name /usr/local/cargo/bin/sccache — the same binary the wrapper exec'd, minus the shell.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n the checkout is too shallow to hold it — intel passed on leftover history, every fresh yoga checkout died 'invalid object name 42be156' 42be156 is on main (the v1.5 spec's squash-merge, 2026-09-05) but deeper than CI's checkout; the long-lived intel workspaces carried it from earlier fetches, the ephemeral yoga runners never do, so guard-tree's 'PP-066 spec v1.6 defect table and its v1.5 RED proof' step exited 128 there (#3097 on yoga-build3). One object fetched by SHA when absent; the table and its RED proof are unchanged. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… abbreviated one, so the shallow-checkout fetch fell through Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n the checkout is too shallow to hold it — intel passed on leftover history, every fresh yoga checkout died 'invalid object name 42be156' 42be156 is on main (the v1.5 spec's squash-merge, 2026-09-05) but deeper than CI's checkout; the long-lived intel workspaces carried it from earlier fetches, the ephemeral yoga runners never do, so guard-tree's 'PP-066 spec v1.6 defect table and its v1.5 RED proof' step exited 128 there (#3097 on yoga-build3). One object fetched by SHA when absent; the table and its RED proof are unchanged. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit fecb358)
… abbreviated one, so the shallow-checkout fetch fell through Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit ba69692)
…red runs never fit 20
This PR's quick-tier part 2 ("every test target that reads the tree") timed out at its own
20-minute cap twice under train load: run 34542045481 on yoga-build2 compiled 811 crates in ~9 min
(a cold per-run target dir, with the shared sccache warm) and was part way through 65,266 tests at the cap,
and attempt 2 on intel-clean-room-15 timed out at 20 minutes as well. The "~6 min" figure was never
reproduced in CI. 45 minutes is still well under the serial tier's 60; the step is re-tightened
once it is measured under load. No guard, test or spec pins the old 20.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ly an ARM box sees, and #2567's parallel Q4_K path was never called The gx10-pool1 canary picked up #3089's ci / lint and failed it (job 103137079500). Lint is clean on x86. On aarch64, `cargo clippy --all-targets -- -D warnings` fails in aprender-compute and aprender-present-terminal, and so does the job's `-p aprender` fallback, because both crates are in the facade's dependency closure. - Four SIMD detectors end with `#[cfg(aarch64)] { return Neon; } Scalar`, which leaves the Scalar tail unreachable on ARM. Each tail is now a cfg'd block. - x86-only helpers were compiled on ARM and never called there: the Q6_K value extractor, the BLIS AVX-512 packers, the prefetch and NT-store thresholds, and the 32x6 packing import. Each now carries the x86_64 cfg of its only callers. - rounding.rs imported the NEON backend and never used it. - Five more detectors, in aprender-core, aprender-serve and trueno-ublk, had the same unreachable tail. They only showed up once the crates above them compiled. - Two x86_64-only BLIS packers in compute.rs are also dead on ARM. The cfg belongs on the functions, but compute.rs already carries 11 complexity violations, and the pre-commit gate refuses any edit to the file until it is decomposed. Until then, `mod compute;` carries a non-x86 `expect(dead_code)` with that reason. `expect` becomes an error as soon as the packers stop being dead. One of the dead functions was a real defect. #2567 made the non-x86 matmul_q4k_f32_parallel actually parallel (its doc records 1.21x on GB10), but the only call site sat inside the x86_64 block, so every ARM machine still ran the serial Q4_K kernel. The dispatcher now calls it on non-x86 above the same 8M-element threshold. A new test checks the threaded path against the serial one on every arch; the coverage module that exercised it only builds on x86_64. Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…er (coverage nightly) Pmat-Ticket: PMAT-1106 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…he nightly coverage run was RED for six days on an environment fact
backends::gpu::device::backward::tests called GpuDevice::new().expect("GPU device") nine times. Under cargo llvm-cov on yoga-build2 (a docker runner with no GPU passthrough) the two silu_backward tests panicked and the whole coverage run exited 2 (run 34575134766); the same class in aprender-cgp is #2848. A missing adapter is an environment fact, not a kernel defect: device_or_skip() prints 'SKIP: no GPU adapter on this host (<err>)' and the test returns. Proven both ways on lambda-vector: with VK_ICD_FILENAMES pointed at nothing, 9 SKIP lines and 9 passed in 0.00 s; with the real adapter, 0 SKIP lines and 9 passed in 1.42 s. The box that has an adapter stays the gate.
Pmat-Ticket: PMAT-1106
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ap resolved to #3089's entry, PMAT-1102's own entry kept Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t in the merge queue and GitHub started no pull_request run for it Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r runs aprender-compute's gpu tests on intel, where there is no adapter; roadmap: PMAT-1106 entry appended Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… run needs --bin apr (apr-cli ships two binaries since 2026-04-18), and M008 asserts the exit code follows the printed verdict instead of assuming a jittered simulated run passes The E1 quick tier (#3089) is the first thing to run aprender-core's falsification_measurement_tests target on a PR: m001 died on 'cargo run could not determine which binary to run' (apr + apr-corpus-ingest), and m008 on 'CI thresholds not met' — cbtop --simulated draws jittered brick timings (3/7 bricks over budget, CV 88 %), so whether thresholds are met is a coin flip. Six cargo-run sites now name --bin apr; M008 asserts exit 0 <=> 'Status: PASS', the contract its name states. Both pass locally on lambda-vector. Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ries + PMAT-1102/PMAT-1106 kept Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
|
Measured from the PVL-001 driver (#3093 waits on this PR: its
So the current gx10-build incarnation can see pmat, and the step's own 🤖 Generated with Claude Code |
…d_project_root stopped at the first Cargo.toml+src/, one level below the workspace; the target was dark in the full tier and the quick tier ran it first (#3112) Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…o-device container (yoga-eph nested CI docker) is an environment fact, not a defect; skip with a cited line instead of failing the quick tier (#3112) Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…_DIR), not the workspace root — the second path assumption in the same dark target; the whole target now runs green locally under nextest (116/116) Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… — anchored on the [workspace] manifest above CARGO_MANIFEST_DIR; the target is 16/16 under nextest (#3126) Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… killed at 20:00 under a 15/16-busy intel (run 34617807644) with every executed test green; the job's 150 remains the cap Pmat-Ticket: PMAT-1102 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rom the crate dir — they could not pass on any box The quick tier (#3089/#3115) ran realizar_integration_tests and toyota_principles_tests for the first time on this PR's workspace-test (run 34620804791, yoga-eph): integration_spec_complete panicked 'Spec file must exist (archived)' and nine toyota rows failed, because every path was written relative to the REPO root while cargo/nextest run tests with the CRATE dir (crates/aprender-core) as cwd. p13/p14 were vacuous for the same reason ('if spec_path.exists()' skipped the body). Repo files now resolve from CARGO_MANIFEST_DIR/../.. (the monorepo_invariants.rs pattern), crate sources (src/audio/stream.rs, src/format/v2.rs, src/audio/mel.rs) from CARGO_MANIFEST_DIR, and p8b judges unsafe_code on the workspace root manifest (the crate inherits [lints] workspace = true). Local: 26/26 + 47/47 under nextest. Pmat-Ticket: PMAT-1102
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
gx10 could not run
ci / lint: clippy errors only an ARM box sees (PMAT-1102)The gx10-pool1 canary picked up #3089's
ci / lintand failed it (job 103137079500). Lint is clean on x86. On aarch64,cargo clippy --all-targets -- -D warnings -A unused-variablesfails, and so does the job's-p aprenderfallback, because the failing crates sit in the facade's dependency closure. pool1 is stopped until this lands (pool2 not started), since every lint job it picked up would fail.What only fails on ARM
#[cfg(aarch64)] { return Neon; } Scalar, so the tail is unreachable on ARM. Locations: aprender-compute ×3, present-terminal, the aprender-core demo, aprender-serve ×2 (one of them a test), trueno-ublk. The tails are now cfg'd blocks.x86_64cfg. The Q6_K chunk function is gatedx86_64 or test.matmul_q4k_f32_parallelactually parallel (its doc records 1.21× on GB10), but its only call site sits inside thex86_64block. Every ARM machine still ran the serial kernel. The dispatcher now calls it on non-x86 above the same 8M-element threshold, andtest_q4k_parallel_matches_serial_on_every_archchecks it against the serial kernel on every arch. The coverage module that exercised it only builds on x86_64.mod compute;carries a non-x86expect(dead_code, reason = …).expectturns into an error as soon as the packers stop being dead. The proper fix is to decompose compute.rs and put the cfg on the functions.Verified locally, with CI's exact lint chain
--target aarch64-unknown-linux-gnu):cargo clippy --all-targets -- -D warnings -A unused-variablesis clean (it had 17 errors). The-p aprenderfallback is clean too.cargo clippy --all-targets -- -D warnings -A unused-variablesis clean.cargo test -p aprender-compute --lib: 3511 passed, plus the new test.check_baseline_ratchets.sh✓,guard_tree.sh --no-cargo✓, exceptcheck_complexity_ratchet.sh, which fails because the local pmat is not the pinned 3.39.0 (instrument, not code).After merge: restart gx10-pool1, then start pool2 (operator P0: aprender jobs run on intel + yoga + gx10).
Pmat-Ticket: PMAT-1102
🤖 Generated with Claude Code