Skip to content

pgw#1014: task rig:gauntlet — every mint shape in one command, plus GPU-REAL cycles on this box - #534

Merged
PaulFidika merged 2 commits into
masterfrom
1014-gauntlet
Aug 7, 2026
Merged

pgw#1014: task rig:gauntlet — every mint shape in one command, plus GPU-REAL cycles on this box#534
PaulFidika merged 2 commits into
masterfrom
1014-gauntlet

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

Every mint shape the fleet runs, in one command, with a table — and the first GPU-real cycles this box has ever done.

task rig:gauntlet

Four variants, each a full production cycle: real child spawn, torch.export + AOTInductor, seal, publish over the real wire, second-process discover + arm + parity. On a model that compiles in under a minute.

Variants declare the outcome they expect, because some exist to demonstrate a refusal. So the gauntlet reports agreement, not pass/fail — and a flip in either direction is news: an expected-red going green means someone fixed something and did not say so. Exit 0 = every variant matched.

variant covers
micro plain lane, 3 entries, container input + trailing plain input
micro-lora lora_bucket=64, 5 entries (adapter fork), the lifted-arm gates (pgw#1001)
micro-4d pgw#998's shape — a 4-D latent with BOTH spatial axes dynamic, so every matmul's M extent is NONLINEAR in the traced symbols. This is z-image's declaration
micro-lora-plain-parent expected RED — a bucket-bearing cell offered to a plain-lane parent, refused at discovery with lane_mismatch before any arm runs

micro-4d being green is a result, not a formality: it re-proves pgw#998's fix on every run instead of trusting a changelog, on the exact shape z-image declares.

micro-lora-plain-parent is the standing form of pgw#999's design question, and the reason attempt 27 must dispatch the adopting pod with the cell's own bucket — that path produces no adopt event at all, which reads exactly like "the cell was never delivered".

GPU-real cycles — task rig:gpu, scripts/rig_gpu_env.sh

An isolated second interpreter with torch 2.13.0+cu126. The approved index was cu128 and it is wrong: cu128 has no torch 2.13.0, only 2.11.0, and torch is an axis verify_declared checks strictly — a cu128 cell would be further from the fleet, not closer. cu126 carries 2.13.0 and its CUDA runs on this box's 12.8 driver. The repo's .venv and the cu130 pin are untouched; the GPU interpreter is selected explicitly and is never a default.

This retires the synthetic-sm caveat. Cells now carry a measured sm_89 — the fleet's own compute capability — and the rig reports covers=plumbing + device (VRAM cap, placement, measured lane).

Carve-out, re-measured on the card before any GPU cycle was counted

card RTX 4070 Laptop, sm_89, 7.63 GiB
budget mint 3.00 + adopt 1.00 = 4.00 GiB (half the card)
cap actually delivered MintRequest.vram_cap_bytes = 3.00 GiB
CardCensus basis=sampled, 7.63 total / 7.46 free — pgw#992's pool bound, live
measured peak 0.011 GiB — 0.4% of the mint cap
GEN_WORKER_HOST_MOVE_GUARD untouched
load gate enforced — refused twice, at 34.5 and 72.8

Two bugs found, one of them in the instrument itself

GPU adopt, unreachable on CPU. A code-only cell binds constants from RESIDENT weights, so a CPU-resident parent adopting a CUDA cell fails inside AOTI itself (update_constant_buffer_func_ ... API call failed). The rig's adopt child is now device-aware.

The gauntlet reported a CPU row as a GPU result. The box hit load 25.6, the rig refused before any leg and exited without writing its json, and run_one read the stale json the CPU run had left at the same path — printing a CPU timing and peak under card=RTX 4070 sm_89. Caught because the row was byte-identical to the CPU run and the table's own synthetic-sm warning fired when it should not have. A green-looking number describing a run that never happened is the defect class this program keeps paying for, and it was in the instrument built to detect it. Fixed three ways: the per-variant json is deleted before each run; a REFUSED exit is its own NOT-RUN state rather than a red (a refusal is a claim about the box, a red is a claim about the mint path); and the table prints n/a.

Not built, with its exact blocker

The w8a8 leg. The lane is reachable now (w8a8_gemm_mode() answers pertensor), but swap_w8a8_linears needs a W8a8Artifact, and quantize_tree_w8a8 needs a diffusers tree (model_index.json + a vocabulary-named denoiser dir) while the micro family writes a flat single-file checkpoint. Eligibility also requires both dims 16-aligned, which in_channels=4 breaks on proj_in/proj_out. Neither is hard; both are real work, and faking either would make the leg test something other than what it claims.

Caveat that rides every GPU number from this box

The cuda identity axis is 12.6, not the fleet's 13.0. torch and sm match exactly; cuda does not — so cells minted here stay local-only. And the parity gate is cosine-based: a uniform ~1% error reads healthy, so it is a directional-degradation instrument, not a max-abs bound.

… GPU-REAL cycles on this box

THE GAUNTLET. Four variants, each a FULL production cycle (real child spawn,
torch.export + AOTInductor, seal, publish over the real wire, second-process
discover + arm + parity), on a model that compiles in under a minute:

  variant                    exp   got               cycle  ent    parity
  micro                      green green OK          15.6s    3  7.15e-07
  micro-lora                 green green OK          29.3s    5  7.15e-07
  micro-4d                   green green OK          27.5s    1  5.96e-07
  micro-lora-plain-parent    red   red   OK          20.4s    5  lane_mismatch=1

Variants declare the outcome they EXPECT, because some exist to demonstrate a
REFUSAL. So the gauntlet reports AGREEMENT, not pass/fail, and a flip in either
direction is news — an expected-red going green means someone fixed something
and did not say so.

`micro-4d` is pgw#998's shape: a 4-D latent with BOTH spatial axes dynamic, so
every matmul's M extent is NONLINEAR in the traced symbols. That is z-image's
declaration. It is GREEN, which re-proves pgw#998's fix on every run instead of
trusting a changelog.

`micro-lora-plain-parent` is the standing form of pgw#999's design question: a
bucket-bearing cell offered to a plain-lane parent is refused at DISCOVERY with
`lane_mismatch`, before any arm runs — A1 step 8's cold-second-pod path.

GPU-REAL CYCLES. `scripts/rig_gpu_env.sh` + `task rig:gpu` build an ISOLATED
second interpreter with torch 2.13.0+cu126 and run `--device cuda`. The
approved index was cu128; it is WRONG — cu128 has no torch 2.13.0, only 2.11.0,
and `torch` is an axis `verify_declared` checks strictly, so a cu128 cell would
be FURTHER from the fleet, not closer. cu126 carries 2.13.0 and its CUDA runs
on this box's 12.8 driver. The repo's .venv and the cu130 pin are untouched.

This RETIRES THE SYNTHETIC-`sm` CAVEAT: cells now carry a measured sm_89 — the
fleet's own compute capability — and the rig reports `covers=plumbing + device
(VRAM cap, placement, measured lane)`.

CARVE-OUT, re-measured on the card BEFORE any GPU cycle was counted: 3.00 GiB
mint + 1.00 GiB adopt = 4.00 of a 7.63 GiB card; the cap really reaches the
child (`vram_cap_bytes = 3.00 GiB`); `CardCensus` reads `basis=sampled` (7.63
total / 7.46 free), so pgw#992's pool bound is live; measured peak 0.011 GiB;
HOST_MOVE_GUARD untouched; the load gate refused twice, at 34.5 and 72.8.

FOUND BY THE GPU LANE IN ITS FIRST CYCLE, unreachable on CPU: a code-only cell
binds constants from RESIDENT weights, so a CPU-resident parent adopting a CUDA
cell fails inside AOTI itself (`update_constant_buffer_func_ ... API call
failed`). The rig's adopt child is now device-aware.

NOT BUILT, with its exact blocker recorded rather than rushed: the w8a8 leg.
The lane is reachable now (`w8a8_gemm_mode()` answers `pertensor`), but
`swap_w8a8_linears` needs a `W8a8Artifact`, and `quantize_tree_w8a8` needs a
DIFFUSERS tree (`model_index.json` + a vocabulary-named denoiser dir) while the
micro family writes a flat single-file checkpoint; and eligibility requires
both dims 16-aligned, which `in_channels=4` breaks on proj_in/proj_out. Neither
is hard; both are real work. See pgw#1014 in the tracker.

CAVEAT that rides every GPU number from this box: the `cuda` identity axis is
12.6, not the fleet's 13.0. `torch` and `sm` match exactly; `cuda` does not, so
these cells stay local-only.
…ts a stale result

The first GPU table reported micro-lora-plain-parent as a GPU row with 20.4s
and peak 0.000 — both its CPU run's numbers. The box hit load 25.6, above the
rig's own gate of 24; the rig refused before any leg and exited WITHOUT writing
its json; run_one then read the stale json the CPU gauntlet had left at the
same path, and the table printed a CPU row under 'card=RTX 4070 sm_89'.

Caught because the row's timing was byte-identical to the CPU run and the
table's own synthetic-sm warning fired when it should not have.

A green-looking number describing a run that never happened is the defect class
this whole program keeps paying for, and it was in the instrument built to
detect it. Three fixes: the per-variant json is deleted before each run, so a
variant that writes none reports 'did-not-run'; the rig's REFUSED exit is its
own NOT-RUN state rather than a red, because a refusal is a claim about the BOX
and a red is a claim about the MINT PATH; and the table prints n/a.
@PaulFidika
PaulFidika merged commit 67627d7 into master Aug 7, 2026
2 checks passed
@PaulFidika
PaulFidika deleted the 1014-gauntlet branch August 7, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant