Skip to content

ci(diet): coverage on tags only, perf-gate table path-scoped, model suites nightly, mutants only on .rs, gate's skipped rule tightened (PMAT-3676) - #3688

Closed
noahgift wants to merge 5 commits into
mainfrom
PMAT-3676-ci-diet
Closed

noahgift wants to merge 5 commits into
mainfrom
PMAT-3676-ci-diet

Conversation

@noahgift

@noahgift noahgift commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Refs #3676 #3668 #3658. Not armed. Per the batching process it rides the next batch. It edits .github/workflows/ (ci.yml, coverage-nightly.yml, guards-nightly.yml). Noah approved all five items directly; the operator's words are quoted on #3676.

keep-open: #3676 stays open until done_when item 1 (≥40% fewer intel runner-minutes per 24 h) is measured after merge.

# Item Change Evidence
1 coverage on tags only skip_coverage: true on the sovereign-ci call. coverage-nightly.yml also runs on push: tags: ['v*'] (COV_FLOOR), with per-ref concurrency. The job gated nothing (no coverage_min) and measured 0 tests (--lib on the facade root, per the existing NOTE). It cost 237 s per run on main 35563537942.
1′ the rmedia trap Every consumer enumerated. The dogfood runs make coverage-check and produces its own number; the steward reacts only to failure; the tests read an embedded string or the Makefile; prepare-release.sh runs llvm-cov locally. None reads CI's number. The one output lost is the advisory Codecov upload. NEW check_coverage_has_producers.sh covers R1–R4 in 10 rows, and each broken link turns RED.
2 book No change. Already path-filtered. book-contracts watches crates/aprender-core/src/** because the chapters run aprender-core code, and check_workflow_path_filters.sh RULE 2 requires that. —
3a perf-gate table The explicit guard-tree step (405 s) is replaced by NEW check_perf_gate_selftest_scoped.sh. It runs only when the diff vs origin/main touches the derived scope: every $ROOT/ path perf_gate.sh reads, followed transitively (8 paths). Otherwise it prints a SUMMARY skip line. An unknown comparand means RUN. It runs whole every night. 11 rows. Mutants that each go RED: not transitive, renames hiding the old side, unknown → skip, prefix-string match.
3b model suites The PR step keeps only falsification_spec_v10_tests, the SATD ratchet vs origin/main and the one property a diff can change. The stress and gpu_state suites run nightly, as they already did. check_model_tests_wired: all 3 still wired.
4 mutants The diff step emits has_rs (--no-renames). Image pull, cargo-mutants install, the run and the upload are gated on it. —
5 gate On a pull_request, mutants must succeed. skipped passes only on other events, where its own if: skips it. The old rule passed a PR whose mutants never ran. NEW check_ci_gate_mutants_rule.sh executes the gate's own marked block for 10 (event, result) pairs. The old rule, planted, goes RED.

Not done, stated: T-4 waiting on the tag's coverage run. That is an autopilot change, deliberately not made on the 0.69 cut day. Today a floor breach reds the tag's coverage run but does not stop the cascade.

Gates:

Update 2026-09-21 ~15:45Z (aprender-62, cop ruling): the ci / gate fix, coverage_on: tag on the merged paiml/.github#77 pin

This PR's first spelling, skip_coverage: true on sovereign-ci@70e51ec, turned ci / gate RED on every PR: that reusable's gate counts a skipped coverage job as a mandatory failure ("coverage did not succeed (result: skipped)"). This PR's own run 35585906113 showed it (it was cancelled unread), and batch-1 #3707 hit it and left this row out. Head a738b926c, a fast-forward from e2d12bdd1 that carries the seated f7b25a6c6 plus:

Proof required before re-folding into batch-2: this PR's own CI green, with the gate logging coverage: NOT MEASURED and roadmap-fragment-parity: NOT-ACTIVE.

keep-open: #3676 stays open until the CI diet is measured on main after this row lands (batch-2); #3668 and #3658 are cited for context and close on their own rows; #3709 is the follow-up migration (stop tracking roadmap.yaml) that arms #73's check.

🤖 Generated with Claude Code

…uites nightly, mutants only on .rs, gate's skipped rule tightened (#3676)

Refs #3676 #3668 #3658

Operator 2026-09-21: aprender releases as fast as possible, intel idle;
"YES, coverage on tags release only." All five workflow items approved by
Noah directly (CLAUDE.md check-in for .github/workflows edits).

1. COVERAGE -- ci.yml `skip_coverage: true` on the sovereign-ci call. That job
   gated nothing (no coverage_min) and measured nothing (`--lib` on the facade
   root: 0 tests, per the existing NOTE), at 237 s per run (main 35563537942).
   coverage-nightly.yml now also triggers on `push: tags: ['v*']` (COV_FLOOR
   via `make coverage`), concurrency per ref so a tag run and the nightly do
   not cancel each other. Every consumer enumerated first (the rmedia trap
   aprender-04 relayed): the pre-publish dogfood's `make coverage-check`
   (produces its own llvm-cov), the queue steward (reacts only to `failure`),
   two tests reading an embedded string / the Makefile, a legacy
   prepare-release.sh (local llvm-cov). None reads CI's number. The one output
   lost: that job's advisory Codecov upload.
   NEW scripts/check_coverage_has_producers.sh asserts the chain: R1 `make -n
   coverage-check` reaches llvm-cov; R2 dogfood runs it (non-comment); R3
   coverage-nightly's `on:` block has schedule AND tags [v*] (comments and
   other top-level keys don't count); R4 numeric COV_FLOOR. 10 rows, each
   link broken -> RED; missing file -> ENV 2.
   NOT done here, stated: making T-4 WAIT on the tag's coverage run is an
   autopilot change, deliberately not made on the 0.69 cut day.

2. BOOK -- no change. book.yml and book-contracts.yml are already path-filtered;
   book-contracts watches crates/aprender-core/src/** because chapter examples
   run aprender-core code, which check_workflow_path_filters.sh RULE 2 requires
   ("a gate that runs code must watch the code it runs"). Narrowing it would
   break that rule.

3a. PERF GATE TABLE -- the explicit guard-tree step (405 s on main 35563537942)
   is replaced by NEW scripts/check_perf_gate_selftest_scoped.sh, dispatched by
   guard_tree.sh --no-cargo: it runs `perf_gate.sh --selftest` only when the
   tree diff vs origin/main touches its DERIVED scope -- every $ROOT/ path
   perf_gate.sh reads, followed transitively through the scripts it calls (8
   paths today) -- and otherwise prints a SUMMARY skip line. Unknown comparand
   or a failed diff -> RUN, never skip. The table runs whole nightly
   (guards-nightly.yml + manifest, 405 s). 11-row case table; mutants RED:
   scope not transitive, renames hiding the old side, unknown -> skip,
   prefix-string matching. Registered in check_no_timing_in_required.sh's
   META_GUARDS (reads no clock).
3b. MODEL SUITES -- the PR step keeps ONLY falsification_spec_v10_tests: it holds
   the SATD ratchet measured against origin/main, the one property here a diff
   can change (and vacuous on the nightly, where HEAD == origin/main). The
   stress and gpu_state suites judge the whole tree; guards-nightly.yml
   already runs all three. check_model_tests_wired: all 3 still wired.

4. MUTANTS -- the diff step emits has_rs (`git diff --no-renames --name-only`);
   image pull, `cargo install cargo-mutants`, the run and the upload are gated
   on it. A diff with no .rs passes quickly instead of installing the tool.

5. GATE -- the mutants rule, between GATE-MUTANTS-RULE markers: on a
   pull_request mutants must SUCCEED (a no-.rs diff succeeds); `skipped` passes
   only on other events, where the job's own `if:` skips it. The old rule
   ("success/skipped both pass") passed a PR whose mutants never ran.
   NEW scripts/check_ci_gate_mutants_rule.sh EXTRACTS that block from ci.yml
   and executes it for 10 (event, result) pairs; its --self-test plants the old
   rule and it goes RED (3 rows); no block -> ENV 2.

Gates: guard_tree --no-cargo: all PASS except check_fleet_pv_shapes_gate.sh,
the known lambda-box fleet-pv condition (identical on main; fixed in #3669);
guard_tree_test 23/0; guards-are-wired, guards-nightly-manifest (8 entries),
no-timing-in-required, model-tests-wired PASS; bashrs 0 errors on the new
files, bashrs-gate PASS; actionlint: the same 7 shellcheck infos as main,
nothing new; roadmap aggregate idempotent; fmt, deny, contracts 1666.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false)

{
 "ticket": "PMAT-3676",
 "head": "e2d12bdd1203666f133be3f766d908950e7f4ee0",
 "width": 3,
 "executor": "agy",
 "agreed": false,
 "auto_merge": {
  "checked": true,
  "was_armed": false,
  "disarmed": false,
  "note": "auto-merge not armed"
 },
 "lanes": [
  {
   "lane": 1,
   "verdict": "FAIL",
   "findings": 3
  },
  {
   "lane": 2,
   "verdict": "FAIL",
   "findings": 3
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 6
  }
 ]
}

…, perf-gate smoke row, gate-rule self-test independent of ci.yml (#3676)

Refs #3676

Round 1 (e2d12bd) was FAIL/FAIL/PASS. The two FAIL lanes held the diff to
the ticket's text and were right on three counts:

1. Item 2 says the book jobs are path-filtered "and nightly always". They were
   filtered, but NO path-filtered workflow had a schedule. book.yml,
   book-contracts.yml and install-script.yml now carry a nightly cron
   (22:30 / 22:45 / 23:00 UTC, ~5h early for the #3292 dispatch delay; book.yml's
   pages deploy stays push-only by its own `if:`).
2. Item 5 says check_workflow_path_filters.sh is updated. It gains RULE 3: a
   path-FILTERED workflow must also trigger on `schedule:` with a cron -- the
   filter is a claim that nothing outside it can break the gate, and the
   nightly is what re-checks it (book.yml sat dark three months, per the
   guard's own header). scripts/lib/workflow_path_filters.py emits a SCHEDULE
   row. Self-test 4 -> 6: no schedule -> RED; `schedule: []` -> RED. On the
   tree before the crons: 3 workflows RED; after: PASS.
3. Item 3 says "The PR path keeps a smoke row" for the perf table. On a skip,
   check_perf_gate_selftest_scoped.sh now requires `perf_gate.sh
   --list-selftests` to exit 0 and enumerate >= 1 case (109 today, ~60 ms), and
   says so on the SUMMARY line. Three end-to-end rows (skip + smoke; in scope
   -> full table; a table that cannot enumerate -> RED). 14 rows; five
   mutants RED (not transitive, renames, unknown -> skip, prefix match, smoke
   removed).
Lane 1's other finding also held: check_ci_gate_mutants_rule.sh checked for
ci.yml before --self-test, so its self-test could not run without the
workflow. The check now sits on the bare-run path only.

Gates: guard_tree --no-cargo all PASS except check_fleet_pv_shapes_gate.sh
(the lambda-box fleet-pv condition, fixed in #3669); guard_tree_test 23/0;
bashrs 0 errors on every changed script, bashrs-gate PASS; actionlint: no
finding beyond main's shellcheck infos.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift

noahgift commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Non-author measured seat: PASS (aprender-7a, 2026-09-21 11:20Z). I did not author PMAT-3676. Everything ran in /mnt/nvme-raid0/agent-wt/seat-3676 (detached, f7b25a6c6), after git fetch origin main.

# command claim measured
1 check_coverage_has_producers.sh / --self-test R1–R4 ok, PASS / 10 rows rc 0: ok R1 (make -n coverage-check reaches llvm-cov), ok R2 (dogfood runs coverage-check), ok R3 (coverage-nightly on schedule AND v* tags), ok R4 (numeric COV_FLOOR), PASS / rc 0 SELF-TEST PASSED: 10 rows ✅
2 check_ci_gate_mutants_rule.sh / --self-test / GATE_RULE_WORKFLOW=/nonexistent … --self-test 10 rows; PR only success passes; off-PR skipped/success pass, failure/cancelled FAIL / old rule RED 3 rows + no-block ENV rc 2 / independent of ci.yml rc 0. Rows: 1 PR success→pass, 2 PR skipped→FAIL, 3 PR cancelled→FAIL, 4 PR failure→FAIL, 5 mg skipped→pass, 6 push skipped→pass, 7 dispatch skipped→pass, 8 mg success→pass, 9 mg failure→FAIL, 10 mg cancelled→FAIL / ok the OLD rule is RED: 3 row(s) + ok a workflow with no rule block is ENV rc=2 / rc 0 SELF-TEST PASSED ✅
3 check_perf_gate_selftest_scoped.sh --self-test / bare 14 rows / SUMMARY skipped + smoke 109, PASS rc 0 SELF-TEST PASSED: 14 rows / SUMMARY perf_gate --selftest skipped: the diff vs origin/main touches 264 path(s), none of the 8 the table reads; guards-nightly.yml runs it whole; smoke: 109 case(s) enumerated, then PASS ✅
4 check_workflow_path_filters.sh --self-test / bare / RULE 3 control (main's book.yml checked out) 6/6 / 3 path-filtered, PASS / FAIL book.yml rc 0 (6/6) / scanned 21 workflow(s), 3 of them path-filtered … PASS … runs nightly / rc 1 FAIL book.yml: path-filtered, but no \schedule:` trigger.`, restored ✅
5 guard_tree.sh --no-cargo all PASS except check_fleet_pv_shapes_gate 79 checks, 1 failed: only check_fleet_pv_shapes_gate.sh [run]. The 3 new guards ran as both [self-test] and [run], all PASS ✅
6 check_no_timing_in_required.sh PASS rc 0 PASS no timing assertion can reach a required status check. ✅
mutation in ci.yml's rule block, pull_request:skipped) ;; before the pull_request:*) arm, then check_ci_gate_mutants_rule.sh rc 1, row 2 FAIL rc 1: FAIL row 2 pull_request skipped wanted 1, got 0. Rows 3/4/9 are still correct. Restored, with 0 tracked changes ✅

noahgift and others added 3 commits September 21, 2026 17:31
… closes-issue --self-test (provisional pin 5997b41)
…-tag

# Conflicts:
#	docs/roadmaps/roadmap.yaml
…— `coverage_on: tag` on the merged #77 pin, not `skip_coverage: true` (PMAT-3676)

`skip_coverage: true` on sovereign-ci@70e51ec made `ci / gate` RED on every
PR: that reusable's gate counts a skipped coverage job as a mandatory failure
("coverage did not succeed (result: skipped)"). #3688's own CI (run
35585906113) and batch-1 #3707 both hit it, and #3707 left this row out.

- sovereign-ci is pinned to paiml/.github f713290c86fcd70d6f26a0faab14e67e6713586f:
  #74's `coverage_on` input, plus #77, which keeps #73's
  roadmap-fragment-parity NOT-ACTIVE while roadmap.yaml is tracked at the
  base (aprender still commits it; #3709 migrates it).
- `coverage_on: tag` replaces `skip_coverage: true`, and ci.yml gets
  `push: tags: ['v*']`, the reusable's second required edit.
- coverage-nightly's v* tag trigger is removed, so ci.yml measures each tag
  once.
- check_coverage_has_producers.sh R3/R5 assert all of this. Self-test 16/16,
  including "skip_coverage: true instead of coverage_on -> RED". On f7b25a6
  it reports R3+R5 RED; after the change it PASSES.
- ci.yml now runs `check_pr_closes_issue.sh --self-test` beside `--body`
  (guard_tree skips it as wired-with-args, so its case table ran in no job;
  found by aprender-f0), and the comment claiming otherwise is corrected.

Local: guard_tree --no-cargo 82/0 (after merging main 225b2a9). The proof
before re-folding into batch-2 is this PR's own CI: green, with the gate
logging `coverage: NOT MEASURED` and `roadmap-fragment-parity: NOT-ACTIVE`.

Refs #3676 #3707 #3709 paiml/.github#77

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3688 head=a738b926c889c40c27c8e85b81c7b3b0e257bcfe verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

@noahgift

Copy link
Copy Markdown
Contributor Author

Superseded — this PR's content is already on main (operator 1-day PR rule triage, 2026-09-24).

Carrying commit: 8c7822f — release(0.69.1): batch 1 (#3772).
Evidence (measured against origin/main @aa7c6ef03): 513/513 added lines present on main; 11/13 files byte-identical, the other 2 (ci.yml, check_no_timing_in_required.sh) were edited further on main after the fold.

Branch is NOT deleted: PMAT-3676-ci-diet @ a738b926c.

@noahgift noahgift closed this Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:code Work is a code change (derived rule, #4159) needs-owner Open work with no live owner — 0.70 batch triage; pick it up and reassign

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant