pgw#992 (P0): "free right now" is not a budget for K children's simultaneous peak - #515
Conversation
…us peak
pgw#868 A4 was right that the pool's per-entry device ask was a guess — 9.9 GiB
of which ~56 % was never observed — and right that the entry children's own
6.02 GiB high-water is the truth. It then divided a MOMENTARY free-VRAM sample
by that truth and called the quotient a simultaneous budget:
'K=2 (vram-bound, goals=mint): 29.5 GiB VRAM (sampled) / 9.9 GiB (estimated) -> 2'
peak_child_device_bytes 6461325312 pool_workers 2 -> 4 peak_concurrency 4
On the first AOT mint ever to reach `inductor_compile` on the real path, that
killed it deterministically at entry 2 of 36:
44.39 GiB card, 2.69 MiB free, OOM on a 14 MiB alloc
9.54 GiB eager-serving parent (resident, by pgw#784's contract)
16.20 GiB mint child's pipeline (resident, this process)
18.61 GiB four entry children
Two facts make the division wrong, and neither is a shortage of card. The free
sample is taken BEFORE the widened children exist, so it prices none of their
growth — its own comment already says it is read "between tenant forwards". And
the two resident consumers keep growing against the same card: 14.9 GiB at the
sample, 25.7 GiB at the OOM.
So the grant is now capped by a budget taken against the CARD:
budget = total
- resident co-tenant (CardCensus, read at pool construction)
- own device high-water (this process, measured)
- tenant reserve (only when a serve goal exists)
`CardCensus` is read once, at construction, and can never be retaken: the
subtraction `total - free - own` names the co-tenant only while the pool's own
children are absent from the card. Every term is an observation; the bound is
recorded whether or not the widen happens, so a future OOM names which term was
wrong. Deliberately NOT a larger DEVICE_RESERVE_BYTES (§4.24) — padding a
constant moves the same unpriced simultaneity onto the next card. An unreadable
census refuses the widen rather than assuming an empty card.
RED/GREEN on the incident's own numbers, driven through the real policy:
master: K 2 -> 4, K*ask + residents = 53.81 GiB vs 44.39 GiB card -> OOM
this: K 2 -> 2, K*ask + residents = 39.77 GiB vs 44.39 GiB card -> FITS
Not a revert: the same measurement on a card that can hold it still widens to
K=4 (asserted). `test_pool_rewiden_pgw868_a4` gains one autouse fixture that
holds the card generous — those rows are about A4's DIVISOR, and the card is
now the other file's variable.
Tests: tests/test_pool_simultaneity_pgw992.py, 9 rows; the A4 file's 16 stay
green. mypy clean (231 files), ruff clean.
…ent about the divisor
The z-image contrast specimen (same `_rewiden` code, a different pod) says the
first version of this fix aimed one notch off:
free_device 16.2 GiB / per_entry 25.0 GiB (ESTIMATED) -> K=1, underwidth=3
That pod's ESTIMATE accidentally protected it, while the L40S died because its
MEASURED peak shrank the denominator. So the fix must not be "prefer the
measured peak" — that is exactly what turned the safe case into the OOM — and
it must not be "distrust the measurement" either. Both are statements about the
DIVISOR, and the divisor was never what was wrong. The invariant is: K
children's simultaneous peak against the residents' future peaks, READ FROM THE
DEVICE, whichever basis supplies the per-entry figure.
That makes it a bound on every width this pool runs, not a patch on the one
path that widens — `entry_workers` divides a momentary free SAMPLE at
construction with precisely the same blind spot. `_apply_simultaneity_bound`
now narrows the constructed width too, floored at 1 (K=1 is the serial path the
pool degrades to; a bound that could forbid it would forbid minting).
On the incident pod the two now compose, visibly:
narrowing the CONSTRUCTED K 2 -> 1 (18.65 GiB of room, 9.90 GiB estimated ask)
capping K 1 -> 2 (A4 asked for 4) (18.65 GiB of room, 7.02 GiB measured ask)
final K = 2, 39.77 GiB vs a 44.39 GiB card -> FITS (master: K=4, 53.81 GiB -> OOM)
The measurement still moves K. It just cannot move it past the card.
Also measured on the z-image pod and the reason this reads the DEVICE rather
than summing what the pool believes is loaded: 16.2 GiB free on an 80 GB card
whose static slot sum is 53.3 GiB — ~9 GiB of CUDA context, allocator
fragmentation and child overhead that no catalog arithmetic can see.
DEFECT FOUND BY THE NEW ROWS: `entry_budget_bytes` returned `-1` for
"unreadable", which collided with a real NEGATIVE budget — a card already
oversubscribed by its residents took the unreadable branch and KEPT its width.
It now returns `None` for unreadable, and an oversubscribed card correctly
falls to the floor of 1.
Tests: 12 rows in test_pool_simultaneity_pgw992.py (three new: the z-image
specimen across both bases, the constructed width bounded, the floor never
breached). Suite 3321 passed, 37 skipped, 1 xfailed. mypy clean, ruff clean.
…e z-image specimen
|
Updated after the z-image contrast specimen (coordinator input, recorded in the pgw#992 filing). The first version capped only The bound is therefore basis-agnostic and applies to every width the pool runs — On the incident pod the two compose: The measurement still moves K. It just cannot move it past the card. Also folded in, as the reason the bound reads the DEVICE rather than summing loaded slots: 16.2 GiB free on an 80 GB card whose static slot sum is 53.3 GiB — ~9 GiB of CUDA context, allocator fragmentation and child overhead no catalog arithmetic can see. Defect the new rows caught in the fix itself: Suite 3321 passed / 37 skipped / 1 xfailed; mypy + ruff clean. 12 rows in Release vehicle: |
1 participant
Fixes the P0 that killed the first AOT mint ever to reach
inductor_compileon the real path, deterministically, at entry 2 of 36.What A4 got right, and the one thing it got wrong
pgw#868 A4 was right that the pool's per-entry device ask was a guess (9.9 GiB, ~56 % never observed) and right that the entry children's own measured high-water (6.02 GiB) is the truth. It then divided a momentary free-VRAM sample by that truth and called the quotient a simultaneous budget:
Two facts make the division wrong, and neither is a shortage of card:
The fix
_rewiden's grant is capped by a budget taken against the card, not against a moment:CardCensusis read once, at construction, and can never be retaken:total − free − ownnames the co-tenant only while the pool's own children are absent from the card. Taken mid-run it would price the pool's own children as co-tenants and narrow forever. That is asserted as a test row.This is the filing's fixes 1 and 2 together — the simultaneous peak is priced, and the serving parent's occupancy is subtracted by measurement rather than assumed static. Explicitly not fix 4:
DEVICE_RESERVE_BYTESis untouched. Padding a constant (§4.24) moves the same unpriced simultaneity onto the next card.Fail-closed everywhere: an unreadable census refuses the widen rather than assuming an empty card.
RED / GREEN — the incident's own numbers, through the real policy
Not a revert. The same measurement on a card that can hold it still widens to K=4 (asserted). The control row asserts that A4's own question, unchanged, still answers 4 — so if the incident ever stops being reproduced, the fix's row stops proving anything.
Acceptance (from the filing)
_rewidenrecords the simultaneity bound it applied — every term, whether or not the widen happens; a refused widen is the interesting rowTests
tests/test_pool_simultaneity_pgw992.py— 9 rows, all on the incident's bytes: the incident reproduced as a control, the bound holding K at 2, a roomy card still widening, every term named, both unreadable-census bases refusing, the tenant reserve as a term of the budget on a serving pod, the census read exactly once, and a driver reading that does not add up yielding no free capacity.tests/test_pool_rewiden_pgw868_a4.pygains one autouse fixture holding the card generous — those 16 rows are about A4's divisor, and the card is now the other file's variable. No A4 assertion changed.Verification
tests/3318 passed, 37 skipped, 1 xfailed (-n 4 --dist loadfile) — no new skipsRelease
Forward-port lands here; the deployable fix rides 0.93.3 off
v0.93.2(package cozy.scheduler;, pre-v1 — master carries the wire-v1 cut and cannot talk to the deployed hub). No collision with #512: that touchescell_key/fleet_cells/aot_cells; this touchesaot_compile_poolonly.