Release process for 0.70+: CRUX-smoke gate on the release binary; full ladder + full CRUX move to nightly; #4033 levers wired into the workflow

[noahgift]

Part of #4033.

Operator words (2026-09-23):

• "lets update our process for .70 and one of your workers on that epic ensure new optimization is part of workflow and we change nightly for the long releaes"
• "anything huge, must be nightly only"

0.69.1 shipped under an OPERATOR EMERGENCY SCOPE: CRUX smoke only. This ticket makes that shape the normal process, with no override needed.

Deliverables:

1. Release gate (normal, not emergency). On the RELEASE BINARY on every GPU host, run CRUX smoke: the certified models × the modes the certification admits, plus the positive control. Add a delta ladder: only the cells whose inference path the diff touches (certification: carry a cell's receipt forward when the diff since its apr_sha doesn't touch that cell's inference path #4037). Also a 1-model per-host smoke plus the MCP surface digest.
2. Nightly (the long run).
   • Full model ladder (all cells, including 27B) plus full-depth CRUX on main, on lambda and gx10, off-hours, under gpu-q (nightly full ladder + CRUX on main; the release sweep runs only the delta since the last green nightly #4040).
   • A release is admitted only if the last green nightly on its merge-base is at most 24 h old; the delta is re-measured.
   • A red nightly files an issue automatically.
   • Coverage moves to nightly as well (Coverage owes 0.69.1: 87.82% vs floor 88, and COVERAGE_EXCLUDE_REGEX has described a different tree since April #3839, fix(coverage): port #3839 to main — exact skips + --ignore-run-fail so coverage-nightly produces a number again #4023).
3. Every Epic: faster release certification — ladder + CRUX (operator: "do all", 2026-09-23) #4033 lever wired into the workflow as it lands. Each must have a must-RED proving it can't turn a real RED green:
   • ladder: run a cell's CPU and CUDA lanes concurrently (the CPU lane doesn't hold the GPU lock) #4034 CPU and CUDA lanes run concurrently;
   • ladder/CRUX: load each model once per cell — one persistent apr serve, every gate as a request #4035 load the model once per cell;
   • CRUX: reference cache keyed by model sha + prompt ids + oracle version; only the apr legs run per host #4036 CRUX reference cache;
   • certification: carry a cell's receipt forward when the diff since its apr_sha doesn't touch that cell's inference path #4037 carry-forward;
   • ladder: CPU==GPU think-block parity on a fixed prefix (e.g. 256 tokens); full 2048-token budget on GPU only #4038 prefix parity;
   • sweep: order cells so CRUX-certified models run first (CRUX can start early), then largest first to cut the tail #4039 cell ordering;
   • nightly full ladder + CRUX on main; the release sweep runs only the delta since the last green nightly #4040 nightly.
4. Update the documents that describe the release:
   • APR-RELEASE-001 (the release spec);
   • the final_sweep runbook;
   • check_model_ladder.sh (the scope becomes a normal mode);
   • the dogfood/release skill;
   • CLAUDE.md's release section.
     Retire phase 2 as a release step.
5. Measure the target: release-night wall time from cut to crates.io, with 0.69.1 as the baseline (freeze 17:00 → publish ~21:00 CEST). Record it per release.

[noahgift]

Prerequisite: #4051. Receipts must stamp per-gate/engine t_start/t_end/lock_wait before any #4033 lever can be judged against the baseline.

[noahgift]

Gate item G-ONT: ONT-001 complete is a MUST-RED for 0.70.0. This is on the operator's instruction, relayed by the cop, verbatim: "the ontology spec MUST be finished in .7". Owner of the gate: aprender-6c [8b6b78]. Rows: infra-83 owns the infra rows; the aprender-side rows ship in 0.70.0, starting with #4047 (the capability-cells extractor), then #3972, #3847 and the rest of the #3269 set.

The rule. 0.70.0 is NO-GO unless every ONT row in infra/docs/specifications/paiml-ontology.md §5, on both the infra and aprender sides, is closed with its done_when measured:

1. scripts/ont/precondition-lint.sh docs/specifications/paiml-ontology.md --ledger <the ONT ledger> reports unbound=0 violations=0. Every row is bound in the ledger to a merged sha.
2. Every row's probe scripts/ont/done_when/ONT-<n>.sh exits 0, run against the release commit (aprender side) and infra main at the cut (infra side). A probe that DECLINEs counts as not done.
3. The infra spec is read at a pinned sha, recorded in the verdict. An untracked or unpinned spec is itself RED (EPIC E0 0.70: Ontology (ONT-001) #3269's v4.3 lesson).

Measured now (infra main @ 29a84779, precondition-lint with no ledger): rows=27, 27 done_when probes. It also reports 3 UNACCOUNTED paths (ONT-4c1, ONT-4b2, ONT-4c3) and one BLOCKED-EXTERNAL: ONT-4c4 waits on aprender#3522 O-1. The bound count needs the ledger, and infra-83 will post it as the baseline. The gate runs the probes; it never quotes a count.

Falsifiers (built with the gate, in the #4045 release-gate PR):

• one fixture row unbound → NO-GO naming it;
• one bound row whose done_when exits 1 → NO-GO naming it;
• a declined probe → NO-GO, never skipped;
• a spec read without a pin → NO-GO;
• 0 rows parsed → decline (the vacuity form).
  Mutants: drop each clause; the named row must go RED.

Blocker to flag now: ONT-4c4 depends on aprender#3522 row O-1 (a real #[kernel] in the workspace). Unless O-1 lands in 0.70.0, G-ONT cannot go green. That is a scope decision for the cop and operator, not something this gate may relax.

[noahgift]

ONT-001 ledger-bound baseline for G-ONT (infra-83, measured 2026-09-23 ~20:05 CEST with scripts/ont/precondition-lint.sh over the spec and docs/audits/ONT-001/ledger.jsonl at each commit, extracted with git archive):

infra commit	rows	bound	unbound	probe_paths	declared	violations	blocked
29a84779 (v4.11 merge)	27	13	14	23	8	0	ONT-4c4 BLOCKED-EXTERNAL on #3522 O-1
1c8e0202 (main now)	27	13	14	23	8	0	same
PMAT-921-v412 (#950, armed)	28	13	15	26	11	0	same; adds ONT-4c5 (this repo's #4047)

Correction to the request's premise: there are no UNACCOUNTED rows at 29a84779 or on main. The "3 UNACCOUNTED (ONT-4c1/4b2/4c3)" shape is exactly what this lint prints when it runs without the ledger: every row then reads as unbound, and bound rows that declare nothing (4c1, 4b2, 4c3) report their tracked paths as UNACCOUNTED. If G-ONT reproduces that, check that its --ledger path resolves.

Bound (13): ONT-0 1 2a 2b 4 4b 4b2 4c1 4c3 6 6b D P. Unbound on v4.12 (15): ONT-E (infra), ONT-11 (pmat), and 13 aprender rows: 4c4 4c5 4c 4d 2c 3a 3b 5 4e 7 8 9 10.
Per-row done_when rc was not run here: the probes need live aprender and pmat worktrees plus pv. The selector's discover run is the measurement for that.

[noahgift]

M8: SHIFT-LEFT, for 0.70 on. Operator, verbatim, relayed by the cop: "all of these trivial issues should be caught hours early and are "fake" gates update .70 to fix and move this release to crates.io". Owner: aprender-6c [8b6b78].

Rule. Every gate that can block a publish runs continuously on the release branch from the freeze onward, on every push to it and at least hourly, against the candidate sha. The branch is already merged with main, so merge-back gates are measured there too. No publish-blocking gate may be first evaluated at the publish step. The measure: 0 gates first-seen-red at publish, counted per release in the ledger (M7).

Classification. Each gate is one of two kinds:

• REAL gates protect users. A red one stops the train (andon).
• BOOKKEEPING gates protect the repo's own accounting. They auto-fix, or report and ticket, but never block the publish. They still block merges to main in normal CI, as today.

gate (0.69.1's late surprise)	class	from 0.70
complexity ratchet vs main (10 fns)	bookkeeping	reported on the branch from freeze; a ticket per function; never blocks publish
census / README counts (1831 vs 1833)	bookkeeping	auto-regenerated and committed on the release branch
bashrs findings (8)	bookkeeping (release scripts, not shipped code)	from freeze; justify or fix in-branch; never blocks publish
pmat-comply CB-200	bookkeeping	report + ticket
claim literals	bookkeeping	regenerated from the measured source
ladder / CRUX scope	real	M4: --scope release, smoke on the release binary + the admitted nightly (#4040), run from freeze
publish preflight R4 (the squash)	real: what is tagged is what was judged	from freeze on the candidate; any rule whose input is not known before the tag is named, and checked at the tag only
clean-room build of the candidate sha	real: reproducible build	dispatched at freeze and on every candidate change, not at publish
dogfood declared gates	real	from freeze on the candidate binary
publish preflight R1–R3, R5–R7	classified rule by rule in the M8 PR, from its rule table	—

Mechanism (proposal): one release-candidate-watch driver on a self-hosted lane (never hosted runners).

• It runs the whole publish-blocking set on the branch head and writes ONE table per run to the release issue: gate, class, verdict, first-red-at.
• Bookkeeping gates run their auto-fix and commit it to the branch. Real gates andon.
• If this needs a .github/workflows change, I check in with the cop first (standing rule). Otherwise it's a forjar timer like nightly full ladder + CRUX on main; the release sweep runs only the delta since the last green nightly #4040's.

Falsifiers:

• a planted bookkeeping red must NOT block the dry-run publish;
• a planted real red MUST;
• a gate added to the publish path without a class is refused (the list is derived from the publish step's own gate invocations, never hand-kept);
• a gate first-seen-red at publish is counted and reported, never hidden.

[noahgift]

M8 (CI hygiene): #4102 — a single #4046 shard left 64G of per-run target on gx10 (infra-d3's measurement). An always() cleanup step, then a size lever chosen by measured kind.

[noahgift]

G-ONT gate built: scripts/check_ont_complete.sh (feat/4045-release-gate-normal). Self-test has 9 rows and 8 mutants, each killed by its named row.

Live baseline (aprender-6c [8b6b78]), with infra main @ b8ee0419e PINNED and the real ONT ledger: RED.

• rows 28, bound 13, UNBOUND 15, violations 0.
• 16 done_when probes are unmet against the release tree: ONT-1, 2c, 3a, 3b, 4c, 4c4, 4c5, 4d, 4e, 5, 7, 8, 9, 10, 11, E.
• BLOCKED-EXTERNAL: ONT-4c4 waits on aprender#3522 O-1 (in 0.70.0 as a hard dependency, per the cop).

0.70.0 cannot pass G-ONT until all 28 rows are bound and every probe exits 0. The gate runs the probes; it never quotes a count.