Skip to content

feat(release): the build ledger's collector lives in the repo — 1091 records committed, no way to produce record 1092 - #3207

Open
noahgift wants to merge 8 commits into
mainfrom
PMAT-1098-build-ledger-collector
Open

noahgift wants to merge 8 commits into
mainfrom
PMAT-1098-build-ledger-collector

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

Closes #3206

APR-RELEASE-001 §3.6.6 makes the ledger the project's memory and §5 P0 wants one record
per gate job. Nothing in the tree wrote one: 1091 records are committed in #3164 as
data, and the only producer was a script in an agent scratchpad. git grep -ln host_class origin/main -- scripts/ .github/ returns nothing.

What lands

file what it is
scripts/collect_build_ledger.sh the collector — same measurement, now in the repo, with a 32-row case table
scripts/check_build_ledger_collector.sh the shim that makes guard_tree.sh run that case table
scripts/shell_lint_baseline.txt 8 → 6, an improvement that is not this PR's (see below)

macos-arm64 is in the instrumented set

Without it, mini's occupancy reads mini=0.0%(0) for ever once #3205 lands — and an
uninstrumented box is indistinguishable from an idle one. host_class is the runner
name's first token (intel-clean-room-3 → intel, mini-m4 → mini), so a new box counts
the day it is named and there is no table to forget.

The case table cost something, and it should be said

The first version had 26 green rows and collected zero records on its first live run.
Every row proved a pure function — record, host_class, clamp_delta, job_slug
and the loop in main() that decides which jobs become records was executed by nothing.
That is this repository's own recurring finding, written by the guard's author, inside
the file about instruments that do not instrument.

So the decision loop is ingest_jobs_tsv, fed a real jobs TSV by the case table and by
main() alike. Its six rows went RED the moment they existed and stayed RED until the
plumbing worked.

Mutation-verified, 4/4 killed:

mutant row that went RED
drop macos-arm64 from the filter filter takes macos-arm64
delete the negative clamp clamp skew -> 0
write 0 instead of null for peak_rss_mb peak_rss_mb null
unanchor the job regex filter skips gpu-quick gate
delete the filter inside ingest_jobs_tsv 3 plumbing rows
delete every _row the shim's vacuity floor

The clamp is not hypothetical: a runner with a skewed clock produced started_at < created_at, and an unclamped negative would subtract from the occupancy total.

Two guards caught this file, which is the point

It was invisible to check_bashrs_gate.sh until git added — that guard's universe is
git ls-files, so an untracked script gets a free pass. Staged, it went RED on two
DET002 and one SC1066:

  • DET002date -u -d "$1" parses the timestamp the API gave us and never reads
    this box's clock. Disable-lined beside the sentence that says so, the idiom already in
    ci_target_watch.sh and llama_bin.sh.
  • SC1066 — bashrs reading jq's $concl=="success" as a shell assignment. The exit
    code is now computed in shell, which is simpler and leaves the jq program data-only.

The baseline change is not mine

shell_lint_baseline.txt 8 → 6. Verified by stashing both new files and re-running:
main is already at 6 while the baseline says 8. Recorded here because this is the PR
touching scripts/, and a shrink-only baseline left unrecorded strands the next one.

Verification

scripts/collect_build_ledger.sh --self-test   → SELF-TEST PASS (32 rows)
scripts/check_build_ledger_collector.sh       → PASS, 32 row(s)   [bare, as guard_tree runs it]
scripts/check_bashrs_gate.sh                  → PASS 297 file(s), 0 SEC/DET/IDEM
scripts/check_shell_lint_ratchet.sh           → PASS (ratcheted)
scripts/check_no_competing_harnesses.sh       → OK
scripts/check_sourced_libs_option_neutral.sh  → OK, 9 libraries
live: --out <tmp> --limit 25                  → 4 records, real fields
      {"host":"gx10-build","host_class":"gx10","job":"workspace-test","queue_wait_s":1159,"exec_s":536}

That queue_wait_s: 1159 is 19 minutes waiting for a runner — the collector's first
real output independently corroborates the fleet saturation measured the same hour.

🤖 Generated with Claude Code

…said "the ledger IS the project memory" and only a scratchpad wrote it

APR-RELEASE-001 §3.6.6 makes the ledger the project's memory, §5 P0 wants one record
per gate job, and §5 refuses to set a build target below 20 records. For the whole
0.67.0 train the only thing producing those records was a script in an agent
scratchpad: 1260 records existed on disk, 1091 were committed in #3164, and nothing
in the tree could produce record 1262. `git grep host_class origin/main` finds
nothing. A memory that dies with the session that wrote it is not memory, and an
instrument that ships no instrument is the defect this spec spends §5 measuring in
other people's jobs.

scripts/collect_build_ledger.sh is that collector, with the measurement unchanged:
queue_wait_s (created→started), exec_s (started→completed), host, host_class, sha,
exit. peak_rss_mb and free_disk_gb stay `null` and stay named in `unmeasured[]` with
a literal [U] — the Actions REST API does not expose them, and a 0 there would be a
peak-RSS average over machines that never reported one.

`macos-arm64` is in the instrumented set. Without it mini's occupancy reads 0.0%(0)
for ever after #3205 lands, and an uninstrumented box is indistinguishable from an
idle one — which is this commit's whole subject, one host further down.

host_class is the runner name's first token (`intel-clean-room-3` → intel, `mini-m4`
→ mini), so a new box is counted the day it is named and there is no table to forget.

WHAT THE CASE TABLE COST, AND WHY IT IS THE SHAPE IT IS

The first version had 26 green rows and collected ZERO records on its first live run.
The rows proved `record`, `host_class`, `clamp_delta`, `job_slug` — every pure
function — and the loop in main() that decides which jobs become records was
executed by nothing. That is this repository's own recurring finding (a guard that
does not scan the surface where the DECISION is made) written by the guard's author,
in the file about instruments that do not instrument.

So the decision loop is `ingest_jobs_tsv`, fed a real jobs TSV by the case table and
by main() alike. Its six rows went RED the moment they existed and stayed RED until
the plumbing worked. Deleting the job filter from inside it turns three of them RED;
that is checked, not asserted.

Mutation-verified, four mutants, all killed: drop `macos-arm64` from the filter →
"filter takes macos-arm64" RED; delete the negative clamp → "clamp skew -> 0" RED
(a runner with a skewed clock produced started_at < created_at, and an unclamped
negative would SUBTRACT from the occupancy total); write 0 instead of null for
peak_rss_mb → "peak_rss_mb null" RED; unanchor the job regex → "filter skips
gpu-quick gate" RED.

check_build_ledger_collector.sh exists because `guard_tree.sh`'s universe is exactly
`git ls-files 'scripts/check_*.sh'` and the collector is a tool, not a guard: its
case table would run nowhere. Naming the tool `check_…` to get it collected would
make guard_tree run a network collector bare in CI. So the tool stays a tool and the
five-line shim is what the tree collects. It carries a vacuity floor — a table that
passes with no rows is the `0 violations over 0 files` signature — and dropping every
row turns it RED while dropping one does not.

The file was also invisible to `check_bashrs_gate.sh` until it was `git add`ed: that
guard's universe is `git ls-files`, so an untracked script gets a free pass. Staged,
it went RED on two DET002 and one SC1066. The DET002 are `date -u -d "$1"`, which
parses the timestamp the API GAVE us and never reads this box's clock — disable-lined
beside the sentence that says so, the idiom already in ci_target_watch.sh. The SC1066
was bashrs reading jq's `$concl=="success"` as a shell assignment; the exit code is
now computed in shell, which is simpler and leaves the jq program data-only.

shell_lint_baseline.txt 8 → 6: that improvement is NOT this PR's. Verified by
stashing these two files and re-running — main is already at 6 while the baseline
says 8. Recorded here because this is the PR touching scripts/ and a shrink-only
baseline left unrecorded strands the next one.

Closes #3206

Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift
noahgift enabled auto-merge September 13, 2026 11:20
…s 74.3% of the fleet's runner-hours and no box could read "80% full"

Measured over four hours of every event, jobs API, completed_at - started_at:

    44.4 runner-hours total
    33.0 h  219 jobs  74.3%  the gate set this file instrumented
    11.4 h  285 jobs  25.7%  invisible — ci / coverage 2.17 h, vendored-schemas 1.65 h,
                             cuda-unit 1.02 h, ci / security, pr-review-shadow, mutants,
                             ci / provenance, gpu-touched

§5 P0 asks for a record per GATE job and the first draft wrote only those, so a quarter
of the work vanished from every occupancy figure derived from the ledger. The operator's
rule is "yoga+gx10 ~80% full under intel queue pressure"; against a numerator missing a
quarter of the hours, 80% was not reachable no matter how full the boxes were.

This is the numerator half of a defect whose denominator half was fixed in the packer the
same hour: CAP counted ALL runners (16/6/5) when only CLEAN-ROOM runners (15/4/3) can take
a PR job, charging gx10 and yoga for gx10-blackwell, yoga-eph, yoga-gpu and
intel-clean-room-16. Same ledger, same hour, gx10 read 37.7% with both defects and 56.5%
with the denominator alone corrected. Neither half is the whole fix and shipping one
without the other would have moved the number while leaving it wrong.

So every completed job gets a record, and `gate_job` is now a FIELD. A reader that wants
§5 P0's gate set filters on it; nobody loses the rest. Live on the same window: 4 records
before, 43 after.

The classifier regex is unchanged and still anchored — unanchored, `gate` also matches
`gpu-quick … gate` and `guard-tree` — it just decides a field instead of deciding whether
a measurement exists at all.

Case table 32 → 36 rows. `present recorded` and `present not a gate job` are the two that
state the correction, and re-introducing the filter inside `ingest_jobs_tsv` turns five
rows RED, so the fix is verified by reverting it rather than by reading it.

Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Contributor Author

Update: the numerator was wrong too, and it is fixed here

After opening this PR I measured what the instrumented set actually covers. Four hours,
every event, jobs API, completed_at − started_at:

runner-hours jobs share
the gate set this PR instrumented 33.0 219 74.3%
invisible to the ledger 11.4 285 25.7%

The missing quarter is ci / coverage (2.17 h), vendored-schemas (1.65 h), cuda-unit
(1.02 h), ci / security, pr-review-shadow, mutants, ci / provenance, gpu-touched.

The operator's standing rule is "yoga+gx10 ~80% full under intel queue pressure".
Against a numerator missing a quarter of the hours, 80% was not reachable no matter how
full the boxes were.

It pairs with a denominator defect fixed the same hour

The packer divided by CAP={"intel":16,"gx10":6,"yoga":5} — the all-runners count —
when only clean-room runners (15/4/3) can take a PR job. It was charging gx10 and yoga
for gx10-blackwell, yoga-eph, yoga-gpu and intel-clean-room-16 (perf-solo): the
four runners already known to be reserved capacity.

box as reported denominator fixed + numerator fixed
intel 16.7% 17.8%
gx10 37.7% 56.5% ~72%
yoga 27.7% 46.2% ~66%

gx10 and yoga carry roughly 3× intel's proportional load. Half a fix would have moved
the number while leaving it wrong, so both halves land.

What changed in the script

gate_job is now a field, not a filter. Every completed job gets a record; a reader
that wants §5 P0's gate set filters on it and nobody loses the rest. Same window, live:
4 records before, 43 after.

The classifier regex is unchanged and still anchored (unanchored, gate also matches
gpu-quick … gate and guard-tree) — it just decides a field instead of deciding whether
a measurement exists.

Case table 32 → 36 rows; present recorded and present not a gate job state the
correction, and re-introducing the filter inside ingest_jobs_tsv turns five rows RED.

scripts/collect_build_ledger.sh --self-test  → SELF-TEST PASS (36 rows)
scripts/check_build_ledger_collector.sh      → PASS, 36 row(s)
scripts/check_bashrs_gate.sh                 → PASS 297 file(s), 0 SEC/DET/IDEM
bashrs lint collect_build_ledger.sh          → 0 error(s)

check_roadmap_diff_additive.sh reported base id(s) missing at head: this branch forked before they landed on main, and a plain update takes one side of the file whole. Resolved as main's roadmap plus this branch's own new entries, each at its sorted slot among same-prefix peers. The other three roadmap guards pass on the result.

Pmat-Ticket: PMAT-1098

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 13, 2026

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3207 head=1955a094df1a839f3c82177ebd5e0392c37c0f43 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

noahgift and others added 2 commits September 13, 2026 14:47
…ver grew while an agent session was running

`check_guards_are_wired.sh` caught it on this PR's own branch:

    FAIL: unwired guards grew 3 -> 4.
      NEW: collect_build_ledger.sh

Nothing in the tree invoked it except its own `--self-test` shim. So the
1091 committed ledger records came from a loop in an agent scratchpad, and
the moment that session ended the ledger stopped. The script's header
already said this out loud —

    "A memory that dies with the session that wrote it is not memory, and
     an instrument that ships no instrument is the same defect this spec
     spends §5 measuring in other people's jobs."

— and then shipped without the caller anyway. This is the third facility
this week that arrived self-tested and uncalled, after
`check_tool_versions.sh --audit-workflow` (#3213) and `cargo_classify.sh`'s
ENV table (#3215). A self-test proves a facility is correct, never that it
is reached.

NIGHTLY, NOT PER-RUN. A collector that commits on every CI run pushes to
the branch that triggered it and triggers itself. The Actions REST API
keeps job timing for 90 days, so once a day loses nothing.

NO FORCE-PUSH, so the branch carries the date: one PR per night, auto-merge
armed, only ever ADDING files under docs/build-ledger/ (records are keyed
sha-host-job, so re-reading a run rewrites the same bytes and the ledger
stays append-only in effect). A night that collected nothing opens no PR.

The commit stages `docs/build-ledger` alone, never `git add -A`: a nightly
holding contents:write must not be able to commit whatever else happened to
be in the tree.

Verified: unwired guards back to 3 (PASS, ratcheted); check_no_hosted_runners
(17 workflows), check_runner_labels, check_workflow_env_defined (17),
check_workflow_path_filters and check_beats_gated all green on the new job.

Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift
noahgift added this pull request to the merge queue Sep 13, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Sep 14, 2026
…his branch was open

One conflict, scripts/shell_lint_baseline.txt. main learned a
`# tool_version=bashrs 7.0.1` header (#3214) after this branch forked;
check_baseline_ratchets.sh now REQUIRES every ratchet baseline to name the
instrument it was recorded under, so a headerless file fails there. Took main's
header; the measured value 6 is the same on both sides.

Verified after the resolve: check_bashrs_gate.sh PASS, 301 files linted, receipt
matched, 0 SEC/DET/IDEM errors.

Pmat-Ticket: PMAT-1098

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift
noahgift enabled auto-merge September 14, 2026 05:24
@noahgift noahgift added this to the 0.68.0 milestone Sep 14, 2026
noahgift added a commit to guyernest/aprender that referenced this pull request Sep 14, 2026
…ssifier existed and nothing asked it (paiml#3216)

* fix(pin): the pv pin called a runner fault a pinning defect — the classifier existed and nothing asked it

guard-cargo turned paiml#3212 red with

    FAIL  pv-pin  the pin failed to resolve pv (rc=1) — a release cannot be
                  decided by a verifier that did not build.

That names a CODE cause. The cause was the host: cargo died with
`could not parse/generate dep info at: …/target/debug/deps/regex-<hash>.d`
/ `No such file or directory (os error 2)`. The proof is in the same job's
own log — the NEXT pv row, 39 s later, built the same pv from the same tree
and passed.

The repo already knew this signature. scripts/cargo_classify.sh ships row C8,
"dep info, and the host had 933G free -> ENV", with a committed fixture named
log_env_dep_info_contention.txt, written precisely so a dep-info death is not
read as ENOSPC and not read as a code defect. The classifier existed, was
self-tested, and the pin never called it. The defect is an unasked rule, not a
missing one.

WHAT CHANGES

  pv_bin.sh            captures its own cargo build, classifies a failure, and
                       prints the shared ENV block instead of "cargo build of
                       aprender-contracts-cli failed".
  verifier_pin.sh      verifier_pin_pv gains rc=3 = "the build did not reach a
                       verdict on this host". rc=1 was answering two different
                       questions with two different responses.
  check_verifier_pinning.sh
                       rows 4/4b/4c report ENV and count the row as UNMEASURED;
                       the summary says so. Additive — the FAIL line still
                       prints, because a run can carry both a runner fault and
                       a real pinning defect.

NOT FAIL-OPEN. Every ENV path still returns non-zero and the job still goes
red: a pin that did not resolve decides nothing, whatever killed it. The only
thing that changes is which cause is named, and therefore whether the response
is "triage the runner and re-run" or "hunt a pinning defect that is not there".

NO RETRY, deliberately. The evidence says one retry would have turned this job
green, which is the argument against it: a retry converts a measurable runner
fault into a slower green, and contention that never reaches a CI verdict is
contention nobody schedules against.

SIX NEW ROWS, and two of them are the plumbing. LIBS / CLASSIFIER / ENV-LOG /
CODE-LOG judge pure functions and would stay green with the pin left entirely
unwired — that is how paiml#3207 shipped 26 green rows that recorded nothing. The
two PIN-RC rows call verifier_pin_pv for real against a throwaway git repo
whose pv_bin.sh reproduces each death, and assert the rc callers switch on.

Mutation-verified, all three killed by the row that should kill them:
  delete the rc=3 arm            -> PIN-RC(env) RED, everything else green
  drop dep-info from the ENV sig -> CLASSIFIER + ENV-LOG + PIN-RC(env) RED
  widen ENV to match /error/     -> CLASSIFIER + CODE-LOG + PIN-RC(code) RED
The third is the fail-open direction and it is caught.

Full guard PASS on this tree; bashrs 0 errors on all three files; the
shell-lint ratchet does not grow.

Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(guard): the ENV fixture pasted a runner's absolute path into a shipped script

check_hardcoded_paths.sh caught it on the first run, correctly: copying the
paiml#3212 failure text verbatim put
/home/noah/data/actions-runner-2/_work/... into scripts/, and the shipped
tier is a DIFFERENTIAL ratchet against main, so one is one too many.

Elided to <workdir>. Nothing is lost — the classifier anchors on cargo's own
framing (`could not parse/generate dep info`, `(os error 2)`) and never on
the path, so all six rows prove exactly what they did before. "Verbatim" was
fidelity to the wrong thing.

Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift
noahgift disabled auto-merge September 14, 2026 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

APR-RELEASE-001 §3.6: nothing in the repo writes the build ledger — 1091 records committed, no way to produce record 1092

1 participant