From fad60f912c2075350e7cfcd6b9187cc932b16e89 Mon Sep 17 00:00:00 2001 From: Paul Fidika Date: Thu, 6 Aug 2026 19:35:29 -0600 Subject: [PATCH 1/3] pgw#992: "free right now" is not a budget for K children's simultaneous peak MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pgw#868 A4 was right that the pool's per-entry device ask was a guess — 9.9 GiB of which ~56 % was never observed — and right that the entry children's own 6.02 GiB high-water is the truth. It then divided a MOMENTARY free-VRAM sample by that truth and called the quotient a simultaneous budget: 'K=2 (vram-bound, goals=mint): 29.5 GiB VRAM (sampled) / 9.9 GiB (estimated) -> 2' peak_child_device_bytes 6461325312 pool_workers 2 -> 4 peak_concurrency 4 On the first AOT mint ever to reach `inductor_compile` on the real path, that killed it deterministically at entry 2 of 36: 44.39 GiB card, 2.69 MiB free, OOM on a 14 MiB alloc 9.54 GiB eager-serving parent (resident, by pgw#784's contract) 16.20 GiB mint child's pipeline (resident, this process) 18.61 GiB four entry children Two facts make the division wrong, and neither is a shortage of card. The free sample is taken BEFORE the widened children exist, so it prices none of their growth — its own comment already says it is read "between tenant forwards". And the two resident consumers keep growing against the same card: 14.9 GiB at the sample, 25.7 GiB at the OOM. So the grant is now capped by a budget taken against the CARD: budget = total - resident co-tenant (CardCensus, read at pool construction) - own device high-water (this process, measured) - tenant reserve (only when a serve goal exists) `CardCensus` is read once, at construction, and can never be retaken: the subtraction `total - free - own` names the co-tenant only while the pool's own children are absent from the card. Every term is an observation; the bound is recorded whether or not the widen happens, so a future OOM names which term was wrong. Deliberately NOT a larger DEVICE_RESERVE_BYTES (§4.24) — padding a constant moves the same unpriced simultaneity onto the next card. An unreadable census refuses the widen rather than assuming an empty card. RED/GREEN on the incident's own numbers, driven through the real policy: master: K 2 -> 4, K*ask + residents = 53.81 GiB vs 44.39 GiB card -> OOM this: K 2 -> 2, K*ask + residents = 39.77 GiB vs 44.39 GiB card -> FITS Not a revert: the same measurement on a card that can hold it still widens to K=4 (asserted). `test_pool_rewiden_pgw868_a4` gains one autouse fixture that holds the card generous — those rows are about A4's DIVISOR, and the card is now the other file's variable. Tests: tests/test_pool_simultaneity_pgw992.py, 9 rows; the A4 file's 16 stay green. mypy clean (231 files), ruff clean. --- changelog.d/pgw992.md | 22 +++ src/gen_worker/aot_compile_pool.py | 200 +++++++++++++++++++++- tests/test_pool_rewiden_pgw868_a4.py | 17 ++ tests/test_pool_simultaneity_pgw992.py | 226 +++++++++++++++++++++++++ 4 files changed, 457 insertions(+), 8 deletions(-) create mode 100644 changelog.d/pgw992.md create mode 100644 tests/test_pool_simultaneity_pgw992.py diff --git a/changelog.d/pgw992.md b/changelog.d/pgw992.md new file mode 100644 index 00000000..7f61ab0a --- /dev/null +++ b/changelog.d/pgw992.md @@ -0,0 +1,22 @@ +- **pgw#992 (P0): the compile pool stops treating a momentary free reading as a + simultaneous budget.** pgw#868 A4 replaced the pool's ~56 %-unobserved + per-entry estimate (9.9 GiB) with the entry children's own MEASURED + high-water (6.02 GiB) and divided the free-VRAM sample by it — `29.5 / 6.02 + -> K=4`. On the first AOT mint ever to reach the compile phase on the real + path that killed the mint deterministically at entry 2 of 36: a 44.39 GiB + L40S holding a 9.54 GiB eager-serving parent (resident by pgw#784's + contract), the mint child's own 16.20 GiB pipeline and four ~6 GiB entry + children — 44.35 of 44.39 GiB, OOM on a 14 MiB allocation. +- The premise was right and the arithmetic was wrong in one way: the free + sample is taken *before* the widened children exist and prices none of their + growth, and the two resident consumers went from 14.9 GiB at the sample to + 25.7 GiB at the OOM. `_rewiden` now caps A4's grant by a budget taken against + the CARD — `total − resident co-tenant − this process's device high-water − + the tenant reserve when a serve goal exists` — with the co-tenant measured by + a `CardCensus` read at pool construction, before child one exists, which is + the only moment that subtraction means anything. +- Every term is an observation and the bound is recorded whether or not the + widen happens, so a future OOM names which term was wrong instead of leaving + a reader to diff two pods that no longer exist. Deliberately NOT a larger + `DEVICE_RESERVE_BYTES` (§4.24): padding a constant moves the same unpriced + simultaneity onto the next card. Unreadable census ⇒ no widen. diff --git a/src/gen_worker/aot_compile_pool.py b/src/gen_worker/aot_compile_pool.py index 3dd14388..70026d4b 100644 --- a/src/gen_worker/aot_compile_pool.py +++ b/src/gen_worker/aot_compile_pool.py @@ -56,7 +56,7 @@ import subprocess import sys import time -from dataclasses import dataclass, field +from dataclasses import dataclass, field, replace from pathlib import Path from typing import ( Any, Callable, Dict, List, Mapping, Optional, Sequence, Tuple) @@ -300,6 +300,85 @@ def facts(self) -> Dict[str, Any]: } +@dataclass(frozen=True) +class CardCensus: + """Who holds the card, taken BEFORE the pool spawns its first child. + + pgw#992: the one reading that makes a simultaneity bound computable. At + pool construction no entry child exists, so everything on the device that + is not this process is, by elimination, the RESIDENT co-tenant — the + eager-serving parent the pgw#784 contract keeps alive through the mint. + Taken later the same subtraction would be meaningless, because the pool's + own children would be inside it. + """ + + total_bytes: int + free_bytes: int + own_reserved_bytes: int + basis: str + + @property + def resident_other_bytes(self) -> int: + """The co-tenant's occupancy. Never negative: a driver that reports + `free + own > total` is reporting something this bound must not turn + into free capacity.""" + return max(0, self.total_bytes - self.free_bytes - self.own_reserved_bytes) + + @property + def readable(self) -> bool: + return self.basis == "sampled" and self.total_bytes > 0 + + def facts(self) -> Dict[str, Any]: + return { + "card_total_bytes": int(self.total_bytes), + "card_free_at_open_bytes": int(self.free_bytes), + "card_own_at_open_bytes": int(self.own_reserved_bytes), + "card_resident_other_bytes": int(self.resident_other_bytes), + "card_census_basis": self.basis, + } + + +def card_census(device: int = -1) -> CardCensus: + """One (total, free, own-reserved) reading of the mint's card. + + All three at the same moment on purpose: the subtraction that names the + co-tenant is only sound if its terms describe one instant. + """ + try: + import torch + + if not torch.cuda.is_available(): + return CardCensus(0, 0, 0, "absent") + dev = torch.cuda.current_device() if device < 0 else int(device) + free, total = torch.cuda.mem_get_info(dev) + return CardCensus( + int(total), int(free), int(torch.cuda.memory_reserved(dev)), + "sampled") + except Exception: # noqa: BLE001 — an unreadable card licenses nothing + return CardCensus(0, 0, 0, "unreadable") + + +def own_device_high_water(device: int = -1) -> int: + """This process's own device high-water (0 = unreadable). + + RESERVED, not allocated — the caching allocator's held blocks are exactly + what a co-resident child cannot have, which is the question a simultaneity + bound asks. The mint child's resident pipeline is the largest single + consumer on the card (16.20 GiB of 44.39 on the pgw#992 pod), and it is the + one consumer this process can measure exactly. + """ + try: + import torch + + if not torch.cuda.is_available(): + return 0 + dev = torch.cuda.current_device() if device < 0 else int(device) + return max(int(torch.cuda.max_memory_reserved(dev)), + int(torch.cuda.memory_reserved(dev))) + except Exception: # noqa: BLE001 + return 0 + + @dataclass(frozen=True) class PoolWidth: """The chosen K and every input that chose it — so a mint's telemetry can @@ -1227,6 +1306,15 @@ def __init__( #: readings — the same question, a measured divisor — so the initial #: row has to survive being superseded. self.width_initial = width + #: pgw#992: who else is on the card, read before the first child exists. + #: `_rewiden` cannot compute a simultaneity bound without it, and a + #: census it cannot read refuses the widen rather than assuming an + #: empty card. + self.census = card_census() + #: The terms of the last simultaneity decision, merged into the width + #: row so a future OOM names WHICH term was wrong instead of leaving a + #: reader to diff two pods that no longer exist. + self.simultaneity: Dict[str, Any] = {} self.peak_concurrency = 0 # pgw#848: the kernel's OOM-kill counter as it stood before this pool # ran. A DELTA over the pool's own wall is evidence; the absolute @@ -1556,7 +1644,8 @@ def _emit_width(self, when: str) -> None: try: from . import activity as activity_mod - facts = self.width.facts() + facts = {**self.width.facts(), **self.census.facts(), + **self.simultaneity} if facts == self._emitted_width_facts: return first = self._emitted_width_facts is None @@ -1624,8 +1713,68 @@ def observe_entry_device(self, report: EntryReport) -> None: self.peak_device_bytes = max(self.peak_device_bytes, peak) self.device_samples += 1 + def entry_budget_bytes(self, ask: int) -> Tuple[int, Dict[str, Any]]: + """Bytes K entry children may hold AT THEIR SIMULTANEOUS PEAKS, with + every term named. ``(budget, terms)``; a budget of ``-1`` means a term + was unreadable and no widen may be granted. + + pgw#992 — the defect this replaces. A4 divided the pool's + free-VRAM SAMPLE by the measured per-entry peak: ``29.5 GiB / 6.02 GiB + -> K=4``. On the real path that killed the first AOT mint ever to reach + the compile phase, deterministically, on entry 2 of 36:: + + 44.39 GiB card, 2.69 MiB free, OOM on a 14 MiB alloc + 9.54 GiB eager-serving parent (resident, by pgw#784's contract) + 16.20 GiB mint child's pipeline (resident, this process) + 18.61 GiB four entry children (4 x ~6 GiB, as measured) + + Two facts make the division wrong, and neither is a shortage of card: + + * **"free right now" is not a simultaneous budget.** The sample is + taken before the widened children exist and prices none of their + growth. Its own comment already says it is read *between tenant + forwards*. + * **The residents keep growing against the same card.** They were + 14.9 GiB at the sample and 25.7 GiB at the OOM. A momentary reading + cannot bound a future peak. + + So the budget is taken against the CARD, not against a moment: + + budget = total + - resident co-tenant (census, measured before child one) + - this process's own device high-water + - the tenant's forward reserve, when a serve goal exists + + Every term is an observation. Deliberately NOT a larger + :data:`DEVICE_RESERVE_BYTES` (§4.24): padding a constant moves the same + unpriced simultaneity somewhere else and re-fires on the next card. + """ + terms: Dict[str, Any] = { + "simultaneity_ask_bytes": int(ask), + "simultaneity_basis": self.census.basis, + } + if not self.census.readable or ask <= 0: + terms["simultaneity_verdict"] = "unreadable — no widen" + return -1, terms + own_peak = max( + own_device_high_water(), int(self.census.own_reserved_bytes)) + reserve = DEVICE_RESERVE_BYTES \ + if WorkerGoals(serve=self.width.serve_goal, + mint=self.width.mint_goal).tenant_reserve_applies() \ + else 0 + budget = (self.census.total_bytes - self.census.resident_other_bytes + - own_peak - reserve) + terms.update({ + "simultaneity_own_peak_bytes": int(own_peak), + "simultaneity_tenant_reserve_bytes": int(reserve), + "simultaneity_budget_bytes": int(budget), + "simultaneity_k_cap": int(max(0, budget) // ask), + }) + return budget, terms + def _rewiden(self) -> None: - """Re-derive K from what the entry children MEASURED (pgw#868 A4). + """Re-derive K from what the entry children MEASURED (pgw#868 A4), + bounded by what the CARD can hold at once (pgw#992). pgw#809 sizes the pool before a single entry has run, and the only per-entry device figure available then is @@ -1644,8 +1793,11 @@ def _rewiden(self) -> None: pgw#847 shape (delete a guess, keep the computation), not a new policy: no reserve moves, no ceiling moves, no bound is invented. - Fail-closed, in five directions: + Fail-closed, in six directions: + * **the grant is capped by the card's simultaneous budget** + (:meth:`entry_budget_bytes`) — the one that was missing, and the one + that killed pgw#868 A1's first real AOT compile; * it never NARROWS. Children are already running against the wider number, and a mid-flight dip is the reading ``device_facts`` takes a max over precisely so it cannot be acted on; @@ -1657,7 +1809,9 @@ def _rewiden(self) -> None: * it re-derives against ``width_initial``'s OWN free-VRAM and host-RAM figures rather than re-probing a card that K running children are sitting on — re-probing would read their footprints as absent - capacity and narrow, which is the opposite of the truth; + capacity and narrow, which is the opposite of the truth. That + argument is right about the FREE figure and was never a licence to + skip the card-wide bound above; * anything raising leaves the width exactly as it was. It changes no artifact. K is not an input to codegen, kernel selection @@ -1679,6 +1833,18 @@ def _rewiden(self) -> None: ask = mint_budget.entry_device_ask(int(self.peak_device_bytes)) if ask <= 0: return + # pgw#992: the cap comes FIRST and is recorded whether or not the + # widen happens — a refused widen is the interesting row. + budget, terms = self.entry_budget_bytes(ask) + self.simultaneity = terms + if budget < 0: + logger.info( + "aot-pool: pgw#992 declining to widen from K=%d — the " + "card census is %s, so K children's simultaneous peak " + "cannot be priced", self.width.workers, self.census.basis) + self._emit_width("simultaneity bound unreadable") + return + k_cap = int(terms["simultaneity_k_cap"]) wider = entry_workers( base.entries, limit=base.limit, @@ -1698,16 +1864,34 @@ def _rewiden(self) -> None: except Exception: # noqa: BLE001 — a re-derivation never fails a mint logger.debug("aot-pool: width re-derivation failed", exc_info=True) return - if wider.workers <= self.width.workers: + granted = min(wider.workers, k_cap) + if granted < wider.workers: + # The row pgw#992 exists to produce: A4's own arithmetic said one + # thing, the card said another, and the card wins BY NAME. + logger.warning( + "aot-pool: pgw#992 capping K %d -> %d (A4 asked for %d) — the " + "card holds %.2f GiB, a %.2f GiB co-tenant and a %.2f GiB " + "resident pipeline, leaving %.2f GiB for entries at %.2f " + "GiB each", + self.width.workers, granted, wider.workers, + self.census.total_bytes / 1024**3, + self.census.resident_other_bytes / 1024**3, + terms["simultaneity_own_peak_bytes"] / 1024**3, + budget / 1024**3, ask / 1024**3) + if granted <= self.width.workers: + # Nothing to grant. Emit anyway: "the pool did NOT widen, and here + # is the bound that stopped it" is the row a later OOM needs. + self._emit_width("simultaneity bound held K") return + wider = replace(wider, workers=granted) logger.info( "aot-pool: pgw#868 A4 K %d -> %d — per-entry device ask measured " "at %.2f GiB over %d entr%s against the %.2f GiB (%s) the pool " - "was sized with", + "was sized with, within a %.2f GiB simultaneous budget (pgw#992)", self.width.workers, wider.workers, ask / 1024**3, self.device_samples, "y" if self.device_samples == 1 else "ies", base.per_entry_device_bytes / 1024**3, - base.per_entry_device_basis) + base.per_entry_device_basis, budget / 1024**3) self.width = wider self.ledger.workers = wider.workers self._emit_width("re-derived from measured entry peaks") diff --git a/tests/test_pool_rewiden_pgw868_a4.py b/tests/test_pool_rewiden_pgw868_a4.py index 65b84697..424fc679 100644 --- a/tests/test_pool_rewiden_pgw868_a4.py +++ b/tests/test_pool_rewiden_pgw868_a4.py @@ -67,6 +67,23 @@ def _sized( goals=worker_goals.MINT_ONLY) +@pytest.fixture(autouse=True) +def _roomy_card(monkeypatch: pytest.MonkeyPatch) -> None: + """Hold the CARD constant and generous for this whole file. + + pgw#992 added a second, independent bound to `_rewiden`: the widened set's + SIMULTANEOUS peak against the card, which on a box with no CUDA is + unreadable and therefore refuses every widen (fail-closed, by design). The + rows in this file are about A4's DIVISOR — an estimate replaced by an + observation — so the card is stated once here and never varies. The + simultaneity bound has its own file + (``test_pool_simultaneity_pgw992.py``), where the card is the variable. + """ + monkeypatch.setattr( + pool, "card_census", + lambda device=-1: pool.CardCensus(512 * _GIB, 512 * _GIB, 0, "sampled")) + + def _report(entry: str, reserved: int) -> pool.EntryReport: return pool.EntryReport( entry=entry, status=pool.COMPILED, diff --git a/tests/test_pool_simultaneity_pgw992.py b/tests/test_pool_simultaneity_pgw992.py new file mode 100644 index 00000000..bc597d78 --- /dev/null +++ b/tests/test_pool_simultaneity_pgw992.py @@ -0,0 +1,226 @@ +"""pgw#992: "free right now" is not a budget for K children's simultaneous peak. + +The run this file is written from — the FIRST AOT mint ever to reach the +compile phase on the real path, dead at entry 2 of 36, deterministically:: + + width_reason 'K=2 (vram-bound, goals=mint): 29.5 GiB VRAM (sampled) + / 9.9 GiB per entry (estimated) -> 2' + peak_child_device_bytes 6461325312 (6.02 GiB, MEASURED) + pool_workers 2 -> 4 peak_concurrency 4 + + OutOfMemoryError: tried to allocate 14.00 MiB; 2.69 MiB free of 44.39 GiB + 9.54 GiB eager-serving parent (resident, pgw#784's contract) + 16.20 GiB mint child's pipeline (resident, this process) + 18.61 GiB four entry children + +A4's premise was right — the 9.9 GiB estimate really was ~56 % unobserved, and +6.02 GiB really is the truth. Its arithmetic divided a MOMENTARY free sample by +that truth and called the quotient a simultaneous budget. It is not one: the +sample was taken before the widened children existed, and the two resident +consumers grew from 14.9 GiB to 25.7 GiB against the same card while it aged. + +Every number below is that pod's. The card is the variable; A4's divisor is +held constant, which is the exact inverse of ``test_pool_rewiden_pgw868_a4``. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from gen_worker import aot_compile_pool as pool +from gen_worker import mint_budget, worker_goals + +_GIB = 1024 ** 3 + +# --- the pod, in bytes ------------------------------------------------------ +CARD_TOTAL = 47661043712 # 44.39 GiB, the L40S as the driver reports it +FREE_AT_OPEN = 31664532480 # 29.49 GiB — the sample A4 divided +SERVING_PARENT = 10243173417 # 9.54 GiB, resident throughout by design +OWN_AT_OPEN = CARD_TOTAL - FREE_AT_OPEN - SERVING_PARENT # 5.36 GiB +OWN_PEAK = 17394617548 # 16.20 GiB — the mint child's pipeline +MEASURED_ENTRY_PEAK = 6461325312 # 6.02 GiB, from a real EntryReport + + +def _incident_census() -> pool.CardCensus: + return pool.CardCensus(CARD_TOTAL, FREE_AT_OPEN, OWN_AT_OPEN, "sampled") + + +def _incident_width() -> pool.PoolWidth: + """K=2 from the REAL policy on the REAL estimate — reproduced, not typed.""" + return pool.entry_workers( + 36, vcpus=256, available_bytes=116 * _GIB, peak_rss_bytes=3 * _GIB, + free_vram_bytes=FREE_AT_OPEN, device_bytes=9.9 * _GIB and int(9.9 * _GIB), + device_basis="estimated", device_lock=True, + goals=worker_goals.MINT_ONLY) + + +def _pool(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, + census: pool.CardCensus, own_peak: int) -> pool.EntryCompilePool: + monkeypatch.setattr(pool, "card_census", lambda device=-1: census) + monkeypatch.setattr(pool, "own_device_high_water", lambda device=-1: own_peak) + return pool.EntryCompilePool(tmp_path / "pool", width=_incident_width()) + + +def _observe(box: pool.EntryCompilePool, reserved: int, n: int = 2) -> None: + for i in range(n): + box.observe_entry_device(pool.EntryReport( + entry=f"unet/dim={i}", status=pool.COMPILED, + peak_device_reserved_bytes=reserved)) + box._rewiden() + + +# --------------------------------------------------------------------------- +# RED: the incident, reproduced and then refused +# --------------------------------------------------------------------------- + +def test_the_free_sample_alone_still_says_K4_which_is_what_went_wrong() -> None: + """The defect, isolated: A4's own question, unchanged, still answers 4. + + This is not a test of the fix — it is the control that proves the fix is + changing the ANSWER and not the question. If this row ever stops saying 4, + the incident is no longer being reproduced and the next row proves nothing. + """ + ask = mint_budget.entry_device_ask(MEASURED_ENTRY_PEAK) + a4 = pool.entry_workers( + 36, vcpus=256, available_bytes=116 * _GIB, peak_rss_bytes=3 * _GIB, + free_vram_bytes=FREE_AT_OPEN, device_bytes=ask, + device_basis="measured", device_lock=True, + goals=worker_goals.MINT_ONLY) + assert a4.workers == 4, a4.reason + # ...and four of them do not fit beside the residents. This is the OOM. + assert 4 * ask + SERVING_PARENT + OWN_PEAK > CARD_TOTAL + + +def test_the_simultaneity_bound_holds_K_at_two_on_the_incident_card( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, +) -> None: + """The fix. Same pod, same measured peak, same A4 arithmetic — and the + pool does not widen, because the CARD cannot hold the widened set.""" + box = _pool(tmp_path, monkeypatch, + census=_incident_census(), own_peak=OWN_PEAK) + assert box.width.workers == 2, box.width.reason + + _observe(box, MEASURED_ENTRY_PEAK) + + assert box.width.workers == 2, ( + "A4 asked for K=4 on a 29.49 GiB free SAMPLE; the card holds 44.39 " + "GiB, of which 25.74 GiB is two resident consumers. Widening here is " + "the OOM.") + ask = mint_budget.entry_device_ask(MEASURED_ENTRY_PEAK) + terms = box.simultaneity + assert terms["simultaneity_budget_bytes"] == ( + CARD_TOTAL - SERVING_PARENT - OWN_PEAK) + assert terms["simultaneity_k_cap"] == 2 + # The granted set actually fits, which is the property that matters. + assert (box.width.workers * ask + SERVING_PARENT + OWN_PEAK) <= CARD_TOTAL + + +def test_the_same_measurement_on_a_card_that_can_hold_it_still_widens( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, +) -> None: + """The bound must not be a revert. Give the identical pool a card with no + co-tenant and a small resident set, and A4's widen goes through.""" + roomy = pool.CardCensus(CARD_TOTAL, CARD_TOTAL - 2 * _GIB, 2 * _GIB, + "sampled") + box = _pool(tmp_path, monkeypatch, census=roomy, own_peak=2 * _GIB) + _observe(box, MEASURED_ENTRY_PEAK) + assert box.width.workers == 4, box.width.reason + assert box.width.per_entry_device_basis == "measured" + + +# --------------------------------------------------------------------------- +# the terms are named, and every unreadable one refuses +# --------------------------------------------------------------------------- + +def test_the_bound_names_every_term_it_applied( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, +) -> None: + """Acceptance row 2: a future OOM must say WHICH term was wrong. + + An unnamed bound is the state this incident was debugged out of — the + width record said ``29.5 GiB / 9.9 GiB -> 2`` and nothing anywhere said + what else was on the card.""" + box = _pool(tmp_path, monkeypatch, + census=_incident_census(), own_peak=OWN_PEAK) + _observe(box, MEASURED_ENTRY_PEAK) + + terms = box.simultaneity + for key in ("simultaneity_ask_bytes", "simultaneity_own_peak_bytes", + "simultaneity_tenant_reserve_bytes", + "simultaneity_budget_bytes", "simultaneity_k_cap", + "simultaneity_basis"): + assert key in terms, key + assert terms["simultaneity_own_peak_bytes"] == OWN_PEAK + assert terms["simultaneity_ask_bytes"] == mint_budget.entry_device_ask( + MEASURED_ENTRY_PEAK) + # ...and the census itself rides the emitted width row. + census = _incident_census().facts() + assert census["card_resident_other_bytes"] == SERVING_PARENT + assert census["card_total_bytes"] == CARD_TOTAL + + +@pytest.mark.parametrize("basis", ["absent", "unreadable"]) +def test_an_unreadable_card_refuses_the_widen( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, basis: str, +) -> None: + """Fail-closed. A pool that cannot price simultaneity does not get to + assume an empty card — the failure mode of guessing here is a 13-minute + mint that dies at entry 2 of 36.""" + box = _pool(tmp_path, monkeypatch, + census=pool.CardCensus(0, 0, 0, basis), own_peak=0) + _observe(box, MEASURED_ENTRY_PEAK) + assert box.width.workers == 2 + assert box.simultaneity["simultaneity_verdict"] == "unreadable — no widen" + + +def test_a_serving_pod_pays_the_tenant_reserve_inside_the_bound( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, +) -> None: + """`serve_goal` is already in the width record, so the pool knows a tenant + forward is coming. The reserve is a TERM of the budget, not a pad bolted + on after the division (§4.24: the bound models the threat).""" + monkeypatch.setattr(pool, "card_census", + lambda device=-1: _incident_census()) + monkeypatch.setattr(pool, "own_device_high_water", lambda device=-1: 0) + serving = pool.entry_workers( + 36, vcpus=256, available_bytes=116 * _GIB, peak_rss_bytes=3 * _GIB, + free_vram_bytes=FREE_AT_OPEN, device_bytes=int(9.9 * _GIB), + device_basis="estimated", device_lock=True, + goals=worker_goals.WorkerGoals(serve=True, mint=True)) + box = pool.EntryCompilePool(tmp_path / "pool", width=serving) + _observe(box, MEASURED_ENTRY_PEAK) + assert box.simultaneity["simultaneity_tenant_reserve_bytes"] == \ + pool.DEVICE_RESERVE_BYTES + assert box.simultaneity["simultaneity_budget_bytes"] == ( + CARD_TOTAL - SERVING_PARENT - OWN_AT_OPEN - pool.DEVICE_RESERVE_BYTES) + + +def test_the_census_is_taken_before_any_child_exists( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, +) -> None: + """Why the census is a CONSTRUCTION-time reading and can never be retaken. + + The subtraction ``total - free - own`` names the co-tenant only while the + pool's own children are absent from the card. Taken mid-run it would price + the pool's own children as co-tenants and narrow forever. + """ + calls: list[int] = [] + + def _census(device: int = -1) -> pool.CardCensus: + calls.append(device) + return _incident_census() + + monkeypatch.setattr(pool, "card_census", _census) + monkeypatch.setattr(pool, "own_device_high_water", lambda device=-1: OWN_PEAK) + box = pool.EntryCompilePool(tmp_path / "pool", width=_incident_width()) + _observe(box, MEASURED_ENTRY_PEAK, n=4) + assert len(calls) == 1, "the census is read once, at construction" + assert box.census.resident_other_bytes == SERVING_PARENT + + +def test_a_driver_reading_that_does_not_add_up_yields_no_free_capacity() -> None: + """`free + own > total` is nonsense, and nonsense must not become room.""" + bad = pool.CardCensus(10 * _GIB, 9 * _GIB, 8 * _GIB, "sampled") + assert bad.resident_other_bytes == 0 From 2a7a466c20b416e68035f21632ec95747f9fc737 Mon Sep 17 00:00:00 2001 From: Paul Fidika Date: Thu, 6 Aug 2026 19:59:22 -0600 Subject: [PATCH 2/3] pgw#992 (2): the bound belongs on EVERY width, and it is not a statement about the divisor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The z-image contrast specimen (same `_rewiden` code, a different pod) says the first version of this fix aimed one notch off: free_device 16.2 GiB / per_entry 25.0 GiB (ESTIMATED) -> K=1, underwidth=3 That pod's ESTIMATE accidentally protected it, while the L40S died because its MEASURED peak shrank the denominator. So the fix must not be "prefer the measured peak" — that is exactly what turned the safe case into the OOM — and it must not be "distrust the measurement" either. Both are statements about the DIVISOR, and the divisor was never what was wrong. The invariant is: K children's simultaneous peak against the residents' future peaks, READ FROM THE DEVICE, whichever basis supplies the per-entry figure. That makes it a bound on every width this pool runs, not a patch on the one path that widens — `entry_workers` divides a momentary free SAMPLE at construction with precisely the same blind spot. `_apply_simultaneity_bound` now narrows the constructed width too, floored at 1 (K=1 is the serial path the pool degrades to; a bound that could forbid it would forbid minting). On the incident pod the two now compose, visibly: narrowing the CONSTRUCTED K 2 -> 1 (18.65 GiB of room, 9.90 GiB estimated ask) capping K 1 -> 2 (A4 asked for 4) (18.65 GiB of room, 7.02 GiB measured ask) final K = 2, 39.77 GiB vs a 44.39 GiB card -> FITS (master: K=4, 53.81 GiB -> OOM) The measurement still moves K. It just cannot move it past the card. Also measured on the z-image pod and the reason this reads the DEVICE rather than summing what the pool believes is loaded: 16.2 GiB free on an 80 GB card whose static slot sum is 53.3 GiB — ~9 GiB of CUDA context, allocator fragmentation and child overhead that no catalog arithmetic can see. DEFECT FOUND BY THE NEW ROWS: `entry_budget_bytes` returned `-1` for "unreadable", which collided with a real NEGATIVE budget — a card already oversubscribed by its residents took the unreadable branch and KEPT its width. It now returns `None` for unreadable, and an oversubscribed card correctly falls to the floor of 1. Tests: 12 rows in test_pool_simultaneity_pgw992.py (three new: the z-image specimen across both bases, the constructed width bounded, the floor never breached). Suite 3321 passed, 37 skipped, 1 xfailed. mypy clean, ruff clean. --- src/gen_worker/aot_compile_pool.py | 115 ++++++++++++++++++----- tests/test_pool_simultaneity_pgw992.py | 121 ++++++++++++++++++++++++- 2 files changed, 211 insertions(+), 25 deletions(-) diff --git a/src/gen_worker/aot_compile_pool.py b/src/gen_worker/aot_compile_pool.py index 70026d4b..029c13db 100644 --- a/src/gen_worker/aot_compile_pool.py +++ b/src/gen_worker/aot_compile_pool.py @@ -1262,6 +1262,16 @@ def __init__( #: pgw#842/th#1359: the width facts as last EMITTED, so a re-emit #: happens only when they actually moved. self._emitted_width_facts: Optional[Dict[str, Any]] = None + #: pgw#992: who else is on the card, read before the first child exists. + #: `_rewiden` cannot compute a simultaneity bound without it, and a + #: census it cannot read refuses the widen rather than assuming an + #: empty card. + self.census = card_census() + #: The terms of the last simultaneity decision, merged into the width + #: row so a future OOM names WHICH term was wrong instead of leaving a + #: reader to diff two pods that no longer exist. + self.simultaneity: Dict[str, Any] = {} + self._apply_simultaneity_bound() self._emit_width("construction") self.inductor_configs = dict(inductor_configs or {}) # pgw#848 item 5: the crash-only half. `bank` is None on every path @@ -1301,20 +1311,12 @@ def __init__( #: How many entry children have contributed one. `_rewiden` refuses to #: act on fewer than :data:`REWIDEN_MIN_SAMPLES`. self.device_samples = 0 - #: The width the pool was CONSTRUCTED with, kept whole. `_rewiden` - #: re-derives against this record's own free-VRAM and host-RAM - #: readings — the same question, a measured divisor — so the initial - #: row has to survive being superseded. - self.width_initial = width - #: pgw#992: who else is on the card, read before the first child exists. - #: `_rewiden` cannot compute a simultaneity bound without it, and a - #: census it cannot read refuses the widen rather than assuming an - #: empty card. - self.census = card_census() - #: The terms of the last simultaneity decision, merged into the width - #: row so a future OOM names WHICH term was wrong instead of leaving a - #: reader to diff two pods that no longer exist. - self.simultaneity: Dict[str, Any] = {} + #: The width the pool was CONSTRUCTED with — AFTER the pgw#992 + #: simultaneity bound, so the record `_rewiden` re-derives against is + #: the one the pool actually ran. `_rewiden` re-uses this record's own + #: free-VRAM and host-RAM readings — the same question, a measured + #: divisor — so the initial row has to survive being superseded. + self.width_initial = self.width self.peak_concurrency = 0 # pgw#848: the kernel's OOM-kill counter as it stood before this pool # ran. A DELTA over the pool's own wall is evidence; the absolute @@ -1713,10 +1715,16 @@ def observe_entry_device(self, report: EntryReport) -> None: self.peak_device_bytes = max(self.peak_device_bytes, peak) self.device_samples += 1 - def entry_budget_bytes(self, ask: int) -> Tuple[int, Dict[str, Any]]: + def entry_budget_bytes( + self, ask: int, + ) -> Tuple[Optional[int], Dict[str, Any]]: """Bytes K entry children may hold AT THEIR SIMULTANEOUS PEAKS, with - every term named. ``(budget, terms)``; a budget of ``-1`` means a term - was unreadable and no widen may be granted. + every term named. ``(budget, terms)``, where ``None`` means a term was + unreadable — distinct from a budget that is readable and NEGATIVE, + which is a card already oversubscribed by its residents and a perfectly + computable answer of "one child, and only because one is the floor". + Conflating the two would let an oversubscribed card take the + unreadable branch and keep whatever width it had. pgw#992 — the defect this replaces. A4 divided the pool's free-VRAM SAMPLE by the measured per-entry peak: ``29.5 GiB / 6.02 GiB @@ -1745,9 +1753,21 @@ def entry_budget_bytes(self, ask: int) -> Tuple[int, Dict[str, Any]]: - this process's own device high-water - the tenant's forward reserve, when a serve goal exists - Every term is an observation. Deliberately NOT a larger - :data:`DEVICE_RESERVE_BYTES` (§4.24): padding a constant moves the same - unpriced simultaneity somewhere else and re-fires on the next card. + Every term is an observation, and every term is read from the DEVICE. + Summing what the pool believes is loaded would not do: on the z-image + pod, 16.2 GiB was free on an 80 GB card whose static slot sum is + 53.3 GiB — ~9 GiB of CUDA context, allocator fragmentation and child + overhead that no catalog arithmetic can see. + + Indifferent to ``ask``'s BASIS, deliberately. The estimate is not safer + than the measurement (it only happened to be larger on one pod) and the + measurement is not more dangerous than the estimate; a bound that + preferred either would be a statement about the divisor, and the + divisor is not what was wrong. + + Deliberately NOT a larger :data:`DEVICE_RESERVE_BYTES` (§4.24): padding + a constant moves the same unpriced simultaneity somewhere else and + re-fires on the next card. """ terms: Dict[str, Any] = { "simultaneity_ask_bytes": int(ask), @@ -1755,7 +1775,7 @@ def entry_budget_bytes(self, ask: int) -> Tuple[int, Dict[str, Any]]: } if not self.census.readable or ask <= 0: terms["simultaneity_verdict"] = "unreadable — no widen" - return -1, terms + return None, terms own_peak = max( own_device_high_water(), int(self.census.own_reserved_bytes)) reserve = DEVICE_RESERVE_BYTES \ @@ -1772,6 +1792,57 @@ def entry_budget_bytes(self, ask: int) -> Tuple[int, Dict[str, Any]]: }) return budget, terms + def _apply_simultaneity_bound(self) -> None: + """Narrow the CONSTRUCTED width to what the card can hold at once. + + pgw#992, second reading. The first version of this fix capped only + ``_rewiden``, which would have held the L40S at K=2 — and the z-image + contrast specimen shows why that is not the invariant. Same + ``_rewiden`` code, a different pod: + + free_device 16.2 GiB / per_entry 25.0 GiB (**estimated**) -> K=1, + underwidth=3 + + The ESTIMATE accidentally protected that pod; the L40S died because the + MEASURED peak shrank the denominator. So the bound cannot be "prefer + the measured peak" or "distrust the measured peak" — either one is a + statement about the DIVISOR, and the divisor is not what was wrong. + **The threat is K children's simultaneous peak against the residents' + future peaks, and it has to be read from the DEVICE regardless of which + basis supplies the per-entry figure.** That makes it a bound on every + width this pool ever runs, not a patch on the one path that widens. + + Measured on the same z-image pod, and the reason this reads the card + rather than adding up what the pool thinks is loaded: 16.2 GiB free on + an 80 GB card whose static slot sum is 53.3 GiB — **~9 GiB of CUDA + context, allocator fragmentation and child overhead that no catalog + arithmetic can see**. + + Never below 1: K=1 is the in-process serial path, it is what the pool + degrades TO, and a bound that could forbid it would forbid minting at + all. Never above what the caller already chose — this only narrows. + """ + ask = int(self.width.per_entry_device_bytes or 0) + budget, terms = self.entry_budget_bytes(ask) + self.simultaneity = terms + if budget is None or ask <= 0: + return + capped = max(1, int(terms["simultaneity_k_cap"])) + if capped >= self.width.workers: + return + logger.warning( + "aot-pool: pgw#992 narrowing the CONSTRUCTED K %d -> %d — the " + "card holds %.2f GiB, a %.2f GiB co-tenant and a %.2f GiB " + "resident set, leaving %.2f GiB for entries at %.2f GiB each", + self.width.workers, capped, self.census.total_bytes / 1024**3, + self.census.resident_other_bytes / 1024**3, + terms["simultaneity_own_peak_bytes"] / 1024**3, + budget / 1024**3, ask / 1024**3) + self.width = replace( + self.width, workers=capped, binding="simultaneity", + reason=(f"K={capped} (simultaneity-bound): {self.width.reason} — " + f"narrowed to what the card holds at once")) + def _rewiden(self) -> None: """Re-derive K from what the entry children MEASURED (pgw#868 A4), bounded by what the CARD can hold at once (pgw#992). @@ -1837,7 +1908,7 @@ def _rewiden(self) -> None: # widen happens — a refused widen is the interesting row. budget, terms = self.entry_budget_bytes(ask) self.simultaneity = terms - if budget < 0: + if budget is None: logger.info( "aot-pool: pgw#992 declining to widen from K=%d — the " "card census is %s, so K children's simultaneous peak " diff --git a/tests/test_pool_simultaneity_pgw992.py b/tests/test_pool_simultaneity_pgw992.py index bc597d78..93d6f039 100644 --- a/tests/test_pool_simultaneity_pgw992.py +++ b/tests/test_pool_simultaneity_pgw992.py @@ -100,14 +100,22 @@ def test_the_simultaneity_bound_holds_K_at_two_on_the_incident_card( pool does not widen, because the CARD cannot hold the widened set.""" box = _pool(tmp_path, monkeypatch, census=_incident_census(), own_peak=OWN_PEAK) - assert box.width.workers == 2, box.width.reason + # The pool OPENS at K=1, not the K=2 the free sample licensed: 18.65 GiB of + # card room cannot hold two children at the 9.9 GiB the pod was still + # ASKING for. That the real pod survived at K=2 is a fact about the + # estimate being wrong, not about two 9.9 GiB children fitting. + assert box.width.workers == 1, box.width.reason + assert box.width.binding == "simultaneity" _observe(box, MEASURED_ENTRY_PEAK) + # ...and the measurement then buys the width back, to the largest K the + # card can actually hold. This is the convergence the bound is for: A4's + # observation still moves K, it just cannot move it past the card. assert box.width.workers == 2, ( "A4 asked for K=4 on a 29.49 GiB free SAMPLE; the card holds 44.39 " - "GiB, of which 25.74 GiB is two resident consumers. Widening here is " - "the OOM.") + "GiB, of which 25.74 GiB is two resident consumers. Widening past 2 " + "here is the OOM.") ask = mint_budget.entry_device_ask(MEASURED_ENTRY_PEAK) terms = box.simultaneity assert terms["simultaneity_budget_bytes"] == ( @@ -224,3 +232,110 @@ def test_a_driver_reading_that_does_not_add_up_yields_no_free_capacity() -> None """`free + own > total` is nonsense, and nonsense must not become room.""" bad = pool.CardCensus(10 * _GIB, 9 * _GIB, 8 * _GIB, "sampled") assert bad.resident_other_bytes == 0 + + +# --------------------------------------------------------------------------- +# the z-image contrast specimen: the bound is not about the DIVISOR +# --------------------------------------------------------------------------- + +# The same `_rewiden` code on a different pod, from the pgw#992 filing: +# free_device 16.2 GiB / per_entry 25.0 GiB (ESTIMATED) -> K=1, underwidth=3 +# and 16.2 GiB free on an 80 GB card whose static slot sum is 53.3 GiB — ~9 GiB +# of CUDA context, allocator fragmentation and child overhead that no catalog +# arithmetic can see. +ZI_CARD_TOTAL = 85899345920 # 80 GB +ZI_FREE = int(16.2 * _GIB) +ZI_PER_ENTRY_ESTIMATE = 25 * _GIB +ZI_STATIC_SLOT_SUM = int(53.3 * _GIB) + + +def test_the_estimate_only_LOOKED_safe_and_the_bound_does_not_rely_on_it( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, +) -> None: + """The invariant, stated as the thing it must NOT be. + + z-image survived because its per-entry ESTIMATE (25.0 GiB) happened to be + larger than the truth; the L40S died because its MEASURED peak (6.02 GiB) + was smaller. A fix that said "prefer the estimate" or "distrust the + measurement" would be a statement about the DIVISOR — and the divisor is + not what was wrong. The bound must bite on the card, whichever basis + supplies the ask. + """ + census = pool.CardCensus( + ZI_CARD_TOTAL, ZI_FREE, ZI_STATIC_SLOT_SUM, "sampled") + monkeypatch.setattr(pool, "card_census", lambda device=-1: census) + monkeypatch.setattr( + pool, "own_device_high_water", lambda device=-1: ZI_STATIC_SLOT_SUM) + + # The card's own reading is 16.2 GiB free where the slot sum says 26.7 GiB + # should be: ~9 GiB is context/fragmentation/child overhead. A bound that + # summed checkpoints instead of reading the device would over-grant by it. + invisible = (ZI_CARD_TOTAL - ZI_STATIC_SLOT_SUM) - ZI_FREE + assert invisible > 9 * _GIB * 0.9 + + room = ZI_CARD_TOTAL - census.resident_other_bytes - ZI_STATIC_SLOT_SUM + for basis, ask in (("estimated", ZI_PER_ENTRY_ESTIMATE), + ("measured", MEASURED_ENTRY_PEAK)): + width = pool.entry_workers( + 4, vcpus=128, available_bytes=256 * _GIB, peak_rss_bytes=3 * _GIB, + free_vram_bytes=ZI_FREE, device_bytes=ask, device_basis=basis, + device_lock=True, goals=worker_goals.MINT_ONLY) + box = pool.EntryCompilePool(tmp_path / f"pool-{basis}", width=width) + granted = box.width.workers + run_ask = int(box.width.per_entry_device_bytes or ask) + # The property, stated so it holds for BOTH bases: the pool never + # grants a second child the card cannot hold. K=1 is exempt because it + # is the floor — and on this pod at the ESTIMATE even one child does + # not fit (25.0 GiB into 15.4 GiB of room), which is the whole point: + # that pod was not protected by a safe policy, it was at the floor. + assert granted >= 1 + if granted > 1: + assert granted * run_ask <= room, ( + f"basis={basis}: granted {granted} children of " + f"{run_ask / _GIB:.2f} GiB into {room / _GIB:.2f} GiB") + assert ZI_PER_ENTRY_ESTIMATE > room, ( + "the z-image estimate does not fit even ONCE — 'the estimate kept it " + "safe' is the floor doing the work, not the policy") + + +def test_the_constructed_width_is_bounded_too_not_only_the_widen( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, +) -> None: + """`_rewiden` is not the only way a pool gets a K it cannot hold. + + `entry_workers` divides a free SAMPLE at construction with exactly the same + blind spot, so the bound belongs on every width the pool runs — not on the + one path that happens to widen. + """ + census = pool.CardCensus(CARD_TOTAL, FREE_AT_OPEN, OWN_AT_OPEN, "sampled") + monkeypatch.setattr(pool, "card_census", lambda device=-1: census) + monkeypatch.setattr(pool, "own_device_high_water", lambda device=-1: OWN_PEAK) + + # A width the free sample licenses (29.49 / 4 GiB -> 7) but the card cannot + # hold beside 25.74 GiB of residents. + optimistic = pool.entry_workers( + 36, vcpus=256, available_bytes=116 * _GIB, peak_rss_bytes=3 * _GIB, + free_vram_bytes=FREE_AT_OPEN, device_bytes=4 * _GIB, + device_basis="measured", device_lock=True, + goals=worker_goals.MINT_ONLY) + assert optimistic.workers >= 5, optimistic.reason + + box = pool.EntryCompilePool(tmp_path / "pool", width=optimistic) + assert box.width.workers == (CARD_TOTAL - SERVING_PARENT - OWN_PEAK) // (4 * _GIB) + assert box.width.binding == "simultaneity" + # `_rewiden` re-derives against the width the pool ACTUALLY ran. + assert box.width_initial.workers == box.width.workers + + +def test_the_bound_never_narrows_below_the_serial_path( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, +) -> None: + """K=1 is what the pool degrades TO. A bound that could forbid it would + forbid minting, which is not a safety property.""" + starved = pool.CardCensus(CARD_TOTAL, 1 * _GIB, CARD_TOTAL - 2 * _GIB, + "sampled") + monkeypatch.setattr(pool, "card_census", lambda device=-1: starved) + monkeypatch.setattr( + pool, "own_device_high_water", lambda device=-1: CARD_TOTAL) + box = pool.EntryCompilePool(tmp_path / "pool", width=_incident_width()) + assert box.width.workers == 1 From 0f25b0d19af4b30f2f7285442e81f7668be804c4 Mon Sep 17 00:00:00 2001 From: Paul Fidika Date: Thu, 6 Aug 2026 19:59:54 -0600 Subject: [PATCH 3/3] pgw#992: changelog fragment covers the construction-time bound and the z-image specimen --- changelog.d/pgw992.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/changelog.d/pgw992.md b/changelog.d/pgw992.md index 7f61ab0a..106cb864 100644 --- a/changelog.d/pgw992.md +++ b/changelog.d/pgw992.md @@ -20,3 +20,17 @@ a reader to diff two pods that no longer exist. Deliberately NOT a larger `DEVICE_RESERVE_BYTES` (§4.24): padding a constant moves the same unpriced simultaneity onto the next card. Unreadable census ⇒ no widen. +- **The bound is on every width, and it is not a statement about the divisor.** + The z-image contrast specimen — same code, `16.2 GiB free / 25.0 GiB per + entry (ESTIMATED) -> K=1` — shows the estimate only *accidentally* protected + that pod, so "prefer the measured peak" is precisely the change that turned + the safe case into the OOM, and "distrust the measurement" would be the same + error mirrored. The constructed width is therefore bounded too (floored at + K=1, the serial path the pool degrades to), and on the incident pod the two + compose: constructed `K 2 -> 1` on the 9.9 GiB estimate, then `K 1 -> 2` once + the 6.02 GiB measurement lands. The measurement still moves K; it cannot move + it past the card. +- Read from the DEVICE, never summed from what the pool believes is loaded: + the same z-image pod showed **16.2 GiB free on an 80 GB card whose static + slot sum is 53.3 GiB** — ~9 GiB of CUDA context, allocator fragmentation and + child overhead that no catalog arithmetic can see.