Skip to content

pgw#1000: the export-parallel gate is deleted, and the number it waited for was the wrong number - #525

Merged
PaulFidika merged 1 commit into
masterfrom
1000-export-parallel
Aug 7, 2026
Merged

pgw#1000: the export-parallel gate is deleted, and the number it waited for was the wrong number#525
PaulFidika merged 1 commit into
masterfrom
1000-export-parallel

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

The finding

GEN_WORKER_AOT_EXPORT_PARALLEL shipped OFF for three releases behind a stated reason:

"Width is bounded by the EXPORT-phase footprint, which nobody has measured... until then this refuses to guess and stays OFF."

The footprint was not unmeasured. It was the wrong number, and nothing said so.

timings["export_peak_device_bytes"] is max_memory_allocated over the entire serial export loop, taken against a resident pipeline — 16,558,897,664 (15.4 GiB) on the only complete 36-entry sdxl AOT mint that has ever finished. So width_for was being asked "how many 15.4 GiB workers fit beside a resident mint child", and 1 was the correct answer, on every card, forever.

A flag guarding a computation with one possible answer is not a safety mechanism — it is a feature that never ran. Its telemetry reported export_parallel_binding = export-footprint-unmeasured on every mint, which reads as "waiting for a measurement" rather than "reading the wrong one".

Why it is worth fixing — the banked numbers

From that same mint (attempt 26, L40S, the first complete 36-entry sdxl AOT mint):

export_s 3891.82    compile_s 9153.12    total_s 9546.65

65 minutes of a 2 h 43 m mint, which pgw#809's K-wide compile pool divides by one. The largest serial term left.

What changes

1. The right quantity is measured. aot_mint._ExportFootprint records one row's delta over the resident baseline — what a worker adds to a card that already holds the module it traces — as per_export_device_bytes. Reported beside the phase high-water, never instead of it: the two size different things (a pool vs. the mint child itself) and conflating them was the defect. The stale key is disconnected, not deprecated — a mint shipping only export_peak_device_bytes now yields width 1 (asserted).

2. No env gate. The width decision lives in code. Width 1 is the natural floor when the budget says so, through named terms.

3. The budget is pgw#992's, unchangedtotal − resident co-tenant − own device high-water. An export worker is priced exactly as an entry-compile child is priced, on the same card, beside the same residents. Export has no claim to a weaker rule than compile: dividing a momentary free-VRAM sample is what killed the first AOT mint to reach the compile phase.

The ordering the budget depends on — found by a test that failed

total − co-tenant − own collapses to exactly free when the census and the own high-water coincide. That is an algebraic identity, so a census read at decision time would price nothing at all. The census is taken before the first row traces, the high-water after, and the gap between them is the resident growth a free reading cannot see (5.36 GiB → 16.20 GiB on the pgw#992 pod). Both the identity and the gap are asserted.

I wrote the first version of that test asserting budget < free unconditionally; it failed, and the failure was the design telling me where the census had to go.

aot_export_reuse is DELETED (591 lines), not re-gated

It could never fire. export_reuse_eligible measured:

family value why
sdxl 0.0 banked, two pods — its Transformer2DModel bakes the sequence length and spatial extents
micro-diffusion 0.0 measured this session on the rig

Two structurally different families, both ineligible, and no family has ever been eligible. GEN_WORKER_AOT_EXPORT_REUSE goes with it, along with its allowlist rows, its named-threat justification and its owned-name census entry. This is the §2.2 answer the env sweep asked for and deferred to a ruling.

Measured on the rig

task rig:micro (3 export entries, container inputs, CPU):

  • cell key ck1-b75141e328c720d3 byte-identical before and after — this re-keys nothing;
  • export_reuse_* and export_parallel_enabled gone from the phase table, export_parallel_budget_bytes added;
  • width still 1 / binding export-footprint-unmeasured on a cardless box — the correct fail-closed answer.

Honest limit: pgw#983 (cu130 build vs a CUDA 12.8 driver) means this box has no CUDA, so the rig cannot exercise a non-zero VRAM divisor. The width arithmetic is proven against attempt 26's real bytes in unit rows; the divisor itself needs a pod, and that rides whichever run Paul authorizes.

Verification

  • tests/ 3399 passed, 37 skipped, 1 xfailed (-n 4 --dist loadfile) — no new skips
  • mypy clean (233 files); ruff clean on src/gen_worker (the 46 in tests/ pre-exist on master, verified)
  • lint_config_reads green — 74 accepted pairs, two fewer than before, no new env conditionals
  • unreached-surface guard unchanged at 19

Follow-on, filed not built

The executor is deliberately not in this PR, and pgw#1000 records why with numbers: the obvious shape (a worker that re-composes the pipeline) cannot exceed width 1 on the very pod that motivates it until the divisor above is read from real hardware. Building it first would be building on the unmeasured assumption that cost attempts 21–24. The tracker carries a materially cheaper alternative — a fused export+compile child, which deletes parent_stage_s 1166.92 + child_program_load_s 1289.59 (41 minutes of pure serialization tax) on top of parallelizing the 65-minute export, because that tax exists only because export and compile happen in different processes.

…ed for was the wrong number

`GEN_WORKER_AOT_EXPORT_PARALLEL` shipped OFF for three releases behind a
stated reason: "width is bounded by the EXPORT-phase footprint, which nobody
has measured... until then this refuses to guess and stays OFF."

The footprint was not unmeasured. `timings["export_peak_device_bytes"]` is
`max_memory_allocated` over the ENTIRE serial export loop, taken against a
resident pipeline — 16,558,897,664 bytes (15.4 GiB) on the only complete
36-entry sdxl AOT mint that has ever finished. `width_for` was therefore being
asked "how many 15.4 GiB workers fit beside a resident mint child", and
answering 1 was correct. A flag guarding a computation with one possible
answer is not a safety mechanism; it is a feature that never ran, and nothing
in its telemetry said so — `export_parallel_binding` reported
`export-footprint-unmeasured` on every mint, which read as "waiting for a
measurement" rather than "reading the wrong one".

WHAT CHANGES

`aot_mint._ExportFootprint` measures one row's DELTA over the resident
baseline — what a worker adds to a card that already holds the module it
traces — and ships it as `per_export_device_bytes`, BESIDE the phase
high-water rather than instead of it. The two size different things (a pool
vs. the mint child itself) and conflating them was the defect. The stale key
is disconnected, not deprecated: a mint shipping only `export_peak_device_bytes`
now yields width 1 (asserted).

The width decision lives in code. No env gate; width 1 is the natural floor
when the budget says so, through named terms. The budget is pgw#992's,
unchanged — total − resident co-tenant − own device high-water — so an export
worker is priced exactly as an entry-compile child is priced, on the same
card, beside the same residents. Export has no claim to a weaker rule.

THE ORDERING THE BUDGET DEPENDS ON, found by a test that failed

`total − co-tenant − own` collapses to exactly `free` when the census and the
own high-water coincide — an algebraic identity, so a census read at decision
time would price nothing at all. The census is taken BEFORE the first row
traces, and the high-water after, and the gap between them is the resident
growth a free reading cannot see (5.36 GiB → 16.20 GiB on the pgw#992 pod).
Both the identity and the gap are asserted.

`aot_export_reuse` (591 lines) IS DELETED, NOT RE-GATED

It could never fire. `export_reuse_eligible` measured 0.0 on sdxl (banked, two
pods) and 0.0 on micro-diffusion (measured this session on the rig) — two
structurally different families, both ineligible, and no family has ever been
eligible. `GEN_WORKER_AOT_EXPORT_REUSE` goes with it, with its allowlist rows,
its named-threat justification and its owned-name census entry.

MEASURED

micro rig (`task rig:micro`, 3 export entries, CPU): cell key
`ck1-b75141e328c720d3` byte-identical before and after — this re-keys nothing.
`export_reuse_*` and `export_parallel_enabled` gone from the phase table;
`export_parallel_budget_bytes` added; width still 1 / binding
`export-footprint-unmeasured` on a cardless box, which is the correct
fail-closed answer (pgw#983: cu130 build vs a CUDA 12.8 driver, so the VRAM
divisor itself needs a pod).

Suite 3399 passed, 37 skipped, 1 xfailed. mypy clean (233 files), ruff clean
on src/gen_worker, config-read guard OK (74 pairs, two fewer), unreached-surface
guard unchanged at 19.
@PaulFidika

Copy link
Copy Markdown
Contributor Author

Rebased onto origin/master (#523, #524 landed). One conflict, and it is more evidence for the deletion: pgw#998 (#524) had to propagate its new aot_shape_hints handoff contract into aot_export_reuse's gate arms — a maintenance tax paid on a feature that has never fired on any family. Resolved by deleting the file, which was the change's whole point; aot_shape_hints keeps its live call site in the compile pool/child.

Re-verified after the rebase: suite 3416 passed, 37 skipped, 1 xfailed; mypy clean (234 files); ruff clean on src/gen_worker; lint_config_reads green at 74 pairs.

@PaulFidika
PaulFidika force-pushed the 1000-export-parallel branch from b65f2c2 to dde7ded Compare August 7, 2026 07:52
@PaulFidika
PaulFidika merged commit c65f8da into master Aug 7, 2026
2 checks passed
@PaulFidika
PaulFidika deleted the 1000-export-parallel branch August 7, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant