pgw#1014: task rig:gauntlet — every mint shape in one command, plus GPU-REAL cycles on this box - #534
Merged
Conversation
… GPU-REAL cycles on this box THE GAUNTLET. Four variants, each a FULL production cycle (real child spawn, torch.export + AOTInductor, seal, publish over the real wire, second-process discover + arm + parity), on a model that compiles in under a minute: variant exp got cycle ent parity micro green green OK 15.6s 3 7.15e-07 micro-lora green green OK 29.3s 5 7.15e-07 micro-4d green green OK 27.5s 1 5.96e-07 micro-lora-plain-parent red red OK 20.4s 5 lane_mismatch=1 Variants declare the outcome they EXPECT, because some exist to demonstrate a REFUSAL. So the gauntlet reports AGREEMENT, not pass/fail, and a flip in either direction is news — an expected-red going green means someone fixed something and did not say so. `micro-4d` is pgw#998's shape: a 4-D latent with BOTH spatial axes dynamic, so every matmul's M extent is NONLINEAR in the traced symbols. That is z-image's declaration. It is GREEN, which re-proves pgw#998's fix on every run instead of trusting a changelog. `micro-lora-plain-parent` is the standing form of pgw#999's design question: a bucket-bearing cell offered to a plain-lane parent is refused at DISCOVERY with `lane_mismatch`, before any arm runs — A1 step 8's cold-second-pod path. GPU-REAL CYCLES. `scripts/rig_gpu_env.sh` + `task rig:gpu` build an ISOLATED second interpreter with torch 2.13.0+cu126 and run `--device cuda`. The approved index was cu128; it is WRONG — cu128 has no torch 2.13.0, only 2.11.0, and `torch` is an axis `verify_declared` checks strictly, so a cu128 cell would be FURTHER from the fleet, not closer. cu126 carries 2.13.0 and its CUDA runs on this box's 12.8 driver. The repo's .venv and the cu130 pin are untouched. This RETIRES THE SYNTHETIC-`sm` CAVEAT: cells now carry a measured sm_89 — the fleet's own compute capability — and the rig reports `covers=plumbing + device (VRAM cap, placement, measured lane)`. CARVE-OUT, re-measured on the card BEFORE any GPU cycle was counted: 3.00 GiB mint + 1.00 GiB adopt = 4.00 of a 7.63 GiB card; the cap really reaches the child (`vram_cap_bytes = 3.00 GiB`); `CardCensus` reads `basis=sampled` (7.63 total / 7.46 free), so pgw#992's pool bound is live; measured peak 0.011 GiB; HOST_MOVE_GUARD untouched; the load gate refused twice, at 34.5 and 72.8. FOUND BY THE GPU LANE IN ITS FIRST CYCLE, unreachable on CPU: a code-only cell binds constants from RESIDENT weights, so a CPU-resident parent adopting a CUDA cell fails inside AOTI itself (`update_constant_buffer_func_ ... API call failed`). The rig's adopt child is now device-aware. NOT BUILT, with its exact blocker recorded rather than rushed: the w8a8 leg. The lane is reachable now (`w8a8_gemm_mode()` answers `pertensor`), but `swap_w8a8_linears` needs a `W8a8Artifact`, and `quantize_tree_w8a8` needs a DIFFUSERS tree (`model_index.json` + a vocabulary-named denoiser dir) while the micro family writes a flat single-file checkpoint; and eligibility requires both dims 16-aligned, which `in_channels=4` breaks on proj_in/proj_out. Neither is hard; both are real work. See pgw#1014 in the tracker. CAVEAT that rides every GPU number from this box: the `cuda` identity axis is 12.6, not the fleet's 13.0. `torch` and `sm` match exactly; `cuda` does not, so these cells stay local-only.
…ts a stale result The first GPU table reported micro-lora-plain-parent as a GPU row with 20.4s and peak 0.000 — both its CPU run's numbers. The box hit load 25.6, above the rig's own gate of 24; the rig refused before any leg and exited WITHOUT writing its json; run_one then read the stale json the CPU gauntlet had left at the same path, and the table printed a CPU row under 'card=RTX 4070 sm_89'. Caught because the row's timing was byte-identical to the CPU run and the table's own synthetic-sm warning fired when it should not have. A green-looking number describing a run that never happened is the defect class this whole program keeps paying for, and it was in the instrument built to detect it. Three fixes: the per-variant json is deleted before each run, so a variant that writes none reports 'did-not-run'; the rig's REFUSED exit is its own NOT-RUN state rather than a red, because a refusal is a claim about the BOX and a red is a claim about the MINT PATH; and the table prints n/a.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every mint shape the fleet runs, in one command, with a table — and the first GPU-real cycles this box has ever done.
task rig:gauntletFour variants, each a full production cycle: real child spawn,
torch.export+ AOTInductor, seal, publish over the real wire, second-process discover + arm + parity. On a model that compiles in under a minute.Variants declare the outcome they expect, because some exist to demonstrate a refusal. So the gauntlet reports agreement, not pass/fail — and a flip in either direction is news: an expected-red going green means someone fixed something and did not say so. Exit 0 = every variant matched.
micromicro-loralora_bucket=64, 5 entries (adapter fork), the lifted-arm gates (pgw#1001)micro-4dmicro-lora-plain-parentlane_mismatchbefore any arm runsmicro-4dbeing green is a result, not a formality: it re-proves pgw#998's fix on every run instead of trusting a changelog, on the exact shape z-image declares.micro-lora-plain-parentis the standing form of pgw#999's design question, and the reason attempt 27 must dispatch the adopting pod with the cell's own bucket — that path produces no adopt event at all, which reads exactly like "the cell was never delivered".GPU-real cycles —
task rig:gpu,scripts/rig_gpu_env.shAn isolated second interpreter with torch 2.13.0+cu126. The approved index was cu128 and it is wrong: cu128 has no torch 2.13.0, only 2.11.0, and
torchis an axisverify_declaredchecks strictly — a cu128 cell would be further from the fleet, not closer. cu126 carries 2.13.0 and its CUDA runs on this box's 12.8 driver. The repo's.venvand the cu130 pin are untouched; the GPU interpreter is selected explicitly and is never a default.This retires the synthetic-
smcaveat. Cells now carry a measuredsm_89— the fleet's own compute capability — and the rig reportscovers=plumbing + device (VRAM cap, placement, measured lane).Carve-out, re-measured on the card before any GPU cycle was counted
MintRequest.vram_cap_bytes = 3.00 GiBCardCensusbasis=sampled, 7.63 total / 7.46 free — pgw#992's pool bound, liveGEN_WORKER_HOST_MOVE_GUARDTwo bugs found, one of them in the instrument itself
GPU adopt, unreachable on CPU. A code-only cell binds constants from RESIDENT weights, so a CPU-resident parent adopting a CUDA cell fails inside AOTI itself (
update_constant_buffer_func_ ... API call failed). The rig's adopt child is now device-aware.The gauntlet reported a CPU row as a GPU result. The box hit load 25.6, the rig refused before any leg and exited without writing its json, and
run_oneread the stale json the CPU run had left at the same path — printing a CPU timing and peak undercard=RTX 4070 sm_89. Caught because the row was byte-identical to the CPU run and the table's own synthetic-smwarning fired when it should not have. A green-looking number describing a run that never happened is the defect class this program keeps paying for, and it was in the instrument built to detect it. Fixed three ways: the per-variant json is deleted before each run; a REFUSED exit is its ownNOT-RUNstate rather than a red (a refusal is a claim about the box, a red is a claim about the mint path); and the table printsn/a.Not built, with its exact blocker
The w8a8 leg. The lane is reachable now (
w8a8_gemm_mode()answerspertensor), butswap_w8a8_linearsneeds aW8a8Artifact, andquantize_tree_w8a8needs a diffusers tree (model_index.json+ a vocabulary-named denoiser dir) while the micro family writes a flat single-file checkpoint. Eligibility also requires both dims 16-aligned, whichin_channels=4breaks onproj_in/proj_out. Neither is hard; both are real work, and faking either would make the leg test something other than what it claims.Caveat that rides every GPU number from this box
The
cudaidentity axis is 12.6, not the fleet's 13.0.torchandsmmatch exactly;cudadoes not — so cells minted here stay local-only. And the parity gate is cosine-based: a uniform ~1% error reads healthy, so it is a directional-degradation instrument, not a max-abs bound.