pgw#1000: the export-parallel gate is deleted, and the number it waited for was the wrong number - #525
Merged
Merged
Conversation
…ed for was the wrong number `GEN_WORKER_AOT_EXPORT_PARALLEL` shipped OFF for three releases behind a stated reason: "width is bounded by the EXPORT-phase footprint, which nobody has measured... until then this refuses to guess and stays OFF." The footprint was not unmeasured. `timings["export_peak_device_bytes"]` is `max_memory_allocated` over the ENTIRE serial export loop, taken against a resident pipeline — 16,558,897,664 bytes (15.4 GiB) on the only complete 36-entry sdxl AOT mint that has ever finished. `width_for` was therefore being asked "how many 15.4 GiB workers fit beside a resident mint child", and answering 1 was correct. A flag guarding a computation with one possible answer is not a safety mechanism; it is a feature that never ran, and nothing in its telemetry said so — `export_parallel_binding` reported `export-footprint-unmeasured` on every mint, which read as "waiting for a measurement" rather than "reading the wrong one". WHAT CHANGES `aot_mint._ExportFootprint` measures one row's DELTA over the resident baseline — what a worker adds to a card that already holds the module it traces — and ships it as `per_export_device_bytes`, BESIDE the phase high-water rather than instead of it. The two size different things (a pool vs. the mint child itself) and conflating them was the defect. The stale key is disconnected, not deprecated: a mint shipping only `export_peak_device_bytes` now yields width 1 (asserted). The width decision lives in code. No env gate; width 1 is the natural floor when the budget says so, through named terms. The budget is pgw#992's, unchanged — total − resident co-tenant − own device high-water — so an export worker is priced exactly as an entry-compile child is priced, on the same card, beside the same residents. Export has no claim to a weaker rule. THE ORDERING THE BUDGET DEPENDS ON, found by a test that failed `total − co-tenant − own` collapses to exactly `free` when the census and the own high-water coincide — an algebraic identity, so a census read at decision time would price nothing at all. The census is taken BEFORE the first row traces, and the high-water after, and the gap between them is the resident growth a free reading cannot see (5.36 GiB → 16.20 GiB on the pgw#992 pod). Both the identity and the gap are asserted. `aot_export_reuse` (591 lines) IS DELETED, NOT RE-GATED It could never fire. `export_reuse_eligible` measured 0.0 on sdxl (banked, two pods) and 0.0 on micro-diffusion (measured this session on the rig) — two structurally different families, both ineligible, and no family has ever been eligible. `GEN_WORKER_AOT_EXPORT_REUSE` goes with it, with its allowlist rows, its named-threat justification and its owned-name census entry. MEASURED micro rig (`task rig:micro`, 3 export entries, CPU): cell key `ck1-b75141e328c720d3` byte-identical before and after — this re-keys nothing. `export_reuse_*` and `export_parallel_enabled` gone from the phase table; `export_parallel_budget_bytes` added; width still 1 / binding `export-footprint-unmeasured` on a cardless box, which is the correct fail-closed answer (pgw#983: cu130 build vs a CUDA 12.8 driver, so the VRAM divisor itself needs a pod). Suite 3399 passed, 37 skipped, 1 xfailed. mypy clean (233 files), ruff clean on src/gen_worker, config-read guard OK (74 pairs, two fewer), unreached-surface guard unchanged at 19.
Contributor
Author
|
Rebased onto Re-verified after the rebase: suite 3416 passed, 37 skipped, 1 xfailed; mypy clean (234 files); ruff clean on |
PaulFidika
force-pushed
the
1000-export-parallel
branch
from
August 7, 2026 07:52
b65f2c2 to
dde7ded
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The finding
GEN_WORKER_AOT_EXPORT_PARALLELshipped OFF for three releases behind a stated reason:The footprint was not unmeasured. It was the wrong number, and nothing said so.
timings["export_peak_device_bytes"]ismax_memory_allocatedover the entire serial export loop, taken against a resident pipeline —16,558,897,664(15.4 GiB) on the only complete 36-entry sdxl AOT mint that has ever finished. Sowidth_forwas being asked "how many 15.4 GiB workers fit beside a resident mint child", and 1 was the correct answer, on every card, forever.A flag guarding a computation with one possible answer is not a safety mechanism — it is a feature that never ran. Its telemetry reported
export_parallel_binding = export-footprint-unmeasuredon every mint, which reads as "waiting for a measurement" rather than "reading the wrong one".Why it is worth fixing — the banked numbers
From that same mint (attempt 26, L40S, the first complete 36-entry sdxl AOT mint):
65 minutes of a 2 h 43 m mint, which pgw#809's K-wide compile pool divides by one. The largest serial term left.
What changes
1. The right quantity is measured.
aot_mint._ExportFootprintrecords one row's delta over the resident baseline — what a worker adds to a card that already holds the module it traces — asper_export_device_bytes. Reported beside the phase high-water, never instead of it: the two size different things (a pool vs. the mint child itself) and conflating them was the defect. The stale key is disconnected, not deprecated — a mint shipping onlyexport_peak_device_bytesnow yields width 1 (asserted).2. No env gate. The width decision lives in code. Width 1 is the natural floor when the budget says so, through named terms.
3. The budget is pgw#992's, unchanged —
total − resident co-tenant − own device high-water. An export worker is priced exactly as an entry-compile child is priced, on the same card, beside the same residents. Export has no claim to a weaker rule than compile: dividing a momentary free-VRAM sample is what killed the first AOT mint to reach the compile phase.The ordering the budget depends on — found by a test that failed
total − co-tenant − owncollapses to exactlyfreewhen the census and the own high-water coincide. That is an algebraic identity, so a census read at decision time would price nothing at all. The census is taken before the first row traces, the high-water after, and the gap between them is the resident growth a free reading cannot see (5.36 GiB → 16.20 GiB on the pgw#992 pod). Both the identity and the gap are asserted.I wrote the first version of that test asserting
budget < freeunconditionally; it failed, and the failure was the design telling me where the census had to go.aot_export_reuseis DELETED (591 lines), not re-gatedIt could never fire.
export_reuse_eligiblemeasured:Transformer2DModelbakes the sequence length and spatial extentsTwo structurally different families, both ineligible, and no family has ever been eligible.
GEN_WORKER_AOT_EXPORT_REUSEgoes with it, along with its allowlist rows, its named-threat justification and its owned-name census entry. This is the §2.2 answer the env sweep asked for and deferred to a ruling.Measured on the rig
task rig:micro(3 export entries, container inputs, CPU):ck1-b75141e328c720d3byte-identical before and after — this re-keys nothing;export_reuse_*andexport_parallel_enabledgone from the phase table,export_parallel_budget_bytesadded;export-footprint-unmeasuredon a cardless box — the correct fail-closed answer.Honest limit: pgw#983 (cu130 build vs a CUDA 12.8 driver) means this box has no CUDA, so the rig cannot exercise a non-zero VRAM divisor. The width arithmetic is proven against attempt 26's real bytes in unit rows; the divisor itself needs a pod, and that rides whichever run Paul authorizes.
Verification
tests/3399 passed, 37 skipped, 1 xfailed (-n 4 --dist loadfile) — no new skipssrc/gen_worker(the 46 intests/pre-exist on master, verified)lint_config_readsgreen — 74 accepted pairs, two fewer than before, no new env conditionalsFollow-on, filed not built
The executor is deliberately not in this PR, and pgw#1000 records why with numbers: the obvious shape (a worker that re-composes the pipeline) cannot exceed width 1 on the very pod that motivates it until the divisor above is read from real hardware. Building it first would be building on the unmeasured assumption that cost attempts 21–24. The tracker carries a materially cheaper alternative — a fused export+compile child, which deletes
parent_stage_s 1166.92 + child_program_load_s 1289.59(41 minutes of pure serialization tax) on top of parallelizing the 65-minute export, because that tax exists only because export and compile happen in different processes.