diff --git a/changelog.d/pgw990.md b/changelog.d/pgw990.md new file mode 100644 index 00000000..f2530a5a --- /dev/null +++ b/changelog.d/pgw990.md @@ -0,0 +1,56 @@ +- **pgw#990: AOT cell adoption was behind a default-OFF pilot flag, and the flag + stopped arriving on 2026-08-05 — so the fleet minted the WRONG KIND of cell and + never looked for one.** `aot_cells.prefer_aot()` + (`Settings.compile_prefer_aot` / `GEN_WORKER_PREFER_AOT`) stood in front of two + call sites, not one: `fleet_cells.enable_compiled`'s discovery branch, **and + `fleet_cells.mint_recipe`'s first line**, which returned `RECIPE_DYNAMO` + whenever it was off. The flag reached pods through a release-scoped + `endpoint_env` entry that had to both match the release (`applies_to`) **and** + be declared by it, and neither held for any release after `bdf54080…`. + Measured on the dev fleet: the last `aot_cell_discovery` event of any kind was + 2026-08-02, three A1 attempts ran with adoption un-armed, and the one cell that + got published carried `kind="torch-inductor-cache"` — a dynamo envelope AOT + discovery rejects by name. pgw#805's docstring had described the loop verbatim + in advance: *"a fleet missed, re-minted the wrong kind (or nothing), and missed + identically on every subsequent pod, forever."* **The gate is DELETED**, not + re-pointed — the Settings field, the env name, the loader mapping, the + predicate and both call sites. `provision`'s lifted-LoRA install now keys off + the artifact's own bucket (it was `bucket and compile_prefer_aot`, so an + adopted bucket-bearing cell reached `aot_serve.enable` UNLIFTED on exactly the + pods the flag had not reached). A pod still handed the retired env name now + reports it as unrecognised owned env, so a stale row is loud rather than + silent. + **Proven on hardware** (pod `zqwhoypxefadug`, L40S, gen-worker 0.93.2): + `self_mint_started/aot … recipe=aot key=ck6-f512ec0e… lane=w8a8-lora64`, the + first `recipe=aot` this program has recorded, alongside the first + `aot_cell_discovery` event of any kind since 2026-08-02. +- **pgw#990: `code_closure` leaves the cell key; `ck5` → `ck6`.** Identity is the + COMPUTATION — traced graph × sm × toolchain × env_seal — and code hashes are a + MEMO, never identity. The ck5 `code_closure` axis was a 147-file source-content + hash, so every wheel release re-keyed every cell in the fleet for edits that + cannot change a traced graph (0.93.0 → 0.93.1 moved the sdxl key on three + plumbing files). It is dropped from the required axes, still RECORDED on every + artifact, and still drives the local re-trace memo. `KEY_SCHEME` is bumped so + older cells strand by name, once, instead of silently re-keying under a reused + prefix. +- **pgw#990: `cell_key.is_key` becomes scheme-AGNOSTIC** — `ck` + 1–2 scheme + digits + 56 hex, byte-identical to tensorhub's `compilecache.IsCellKey` and for + the reason th#1183 gives: pinning the current scheme turned every other-scheme + cell into `unreadable_cell_key`, which is both a lie and a filter no axis + justifies. **This is load-bearing for the ck6 bump above**: without it a worker + built from a ck5-pinned tree refuses every ck6 cell before consulting a single + axis, which would strand the cells minted by 0.93.2 workers. Verified by + running a 0.93.2-shaped declare through the current filter. +- **pgw#990: a gate that changes the recipe must SAY SO.** The deleted flag was + the only one of `mint_recipe`'s paths that returned `RECIPE_DYNAMO` without + calling `_decline`, so it emitted nothing — the sole tell that a whole fleet was + minting the wrong kind was the word `dynamo` inside a SUCCESS event's detail. + An AST guard now asserts that the only `return RECIPE_DYNAMO` inside + `mint_recipe` is the one in `_decline`, which emits before it returns. + + > **Release note.** These changes shipped in the **0.93.2** tag, which was cut + > from `v0.93.1` (pre-`cozy.scheduler.v1`) so the pgw#868 A1 pod run could use + > a wheel that talks to the pre-v1 dev hub. This is how they reach `master`. + > The 0.93.2 tag targets an UNSIGNED commit; that is recorded as a deliberate + > exception in pgw#990's tracker entry rather than fixed by rewriting released + > history. diff --git a/docs/environment.md b/docs/environment.md index d8da6e78..e9d3079b 100644 --- a/docs/environment.md +++ b/docs/environment.md @@ -63,7 +63,6 @@ this page covers the worker itself. | `TENSORHUB_URL` | `tensorhub_url` | standalone-CLI resolve base URL | | `TENSORHUB_CACHE_DIR` | `tensorhub_cache_dir` | THE cache/CAS root knob — move cache/CAS off `/tmp` (cozy local persistence). `cache_paths.tensorhub_cas_dir()` derives the CAS from this and nothing else, so this is also what isolates the standalone `cli/run.py` in tests | | `TENSORHUB_CAS_DIR` | `tensorhub_cas_dir` | narrow override consulted at only two sites (`models/provision.py`, `procsplit/parent.py`). `cache_paths.tensorhub_cas_dir()` IGNORES it — setting it does not move the CAS for most call sites, and does not isolate `cli/run.py` | -| `GEN_WORKER_PREFER_AOT` | `compile_prefer_aot` | prefer AOT compile cells over JIT. Deliberately NOT part of `env_seal`'s sealed config table | | `TENSORHUB_FILL_SOURCE_DIR` | `tensorhub_fill_source_dir` | th#850 managed-tier ruling: an endpoint-scoped datacenter-warm CAS mount (RunPod volume), checked before R2 on a blob miss and write-through warmed from R2. Never the CAS root — that always stays `TENSORHUB_CACHE_DIR`/local. tensorhub sets this only when a volume is attached; ismount-guarded, so a plain directory never gets mistaken for it | ## C2PA Content Credentials (Settings fields, th#714) diff --git a/src/gen_worker/aot_cells.py b/src/gen_worker/aot_cells.py index 5d71113e..ab907a90 100644 --- a/src/gen_worker/aot_cells.py +++ b/src/gen_worker/aot_cells.py @@ -1,4 +1,4 @@ -"""AOT cell discovery — the pgw#722 pilot's F1 seam (flag-gated, default OFF). +"""AOT cell discovery — the F1 seam every serving pod runs (pgw#722, pgw#990). A serving pod cannot COMPUTE an exported cell's key: an ``aot-inductor`` cell is STAMPED at mint with axes only the mint holds (its traced-graph @@ -6,9 +6,9 @@ never name it and the hub's worker-owned pull-by-key delivery (th#883) never lights it up. Published AOT cells are therefore provably dark. -This module is the minimal pilot delivery: **fetch-and-filter**. At arm -time (``fleet_cells.enable_compiled``, behind ``Settings. -compile_prefer_aot``) the worker lists the family cell repo's checkpoints +This module is **fetch-and-filter**. At arm time +(``fleet_cells.enable_compiled``, unconditionally since pgw#990) the worker +lists the family cell repo's checkpoints through the hub's existing catalog read API, filters for a cell THIS runtime can serve, downloads the artifact, and feeds it into the existing ``provision.enable_compiled`` HIT path — where the pgw#709 receipt gate @@ -57,8 +57,6 @@ from . import boot_phases as boot_mod from . import compile_cache as cc from .procsplit import broker -from . import config -from .config import Settings from .models.chunk_cas import sha256_file from .models.chunk_cas import ( CAS_CHUNK_SIZE_BYTES, @@ -97,15 +95,6 @@ }) -def prefer_aot(settings: Optional[Settings] = None) -> bool: - """The pilot flip switch (typed pod-launch knob, default OFF). - - Takes the `Settings` when the caller has them; otherwise reads the ones the - process entry installed (§1.18 — never the environment). - """ - return bool((settings or config.current()).compile_prefer_aot) - - @dataclass(frozen=True) class AdoptedAotCell: """Identity of one discovered, downloaded, runtime-matched AOT cell. @@ -505,5 +494,4 @@ def _discover_inner( "AdoptedAotCell", "EVENT", "discover", - "prefer_aot", ] diff --git a/src/gen_worker/aot_mint.py b/src/gen_worker/aot_mint.py index 6287f7c0..cd57796f 100644 --- a/src/gen_worker/aot_mint.py +++ b/src/gen_worker/aot_mint.py @@ -2878,8 +2878,6 @@ class hash folds that entry's ``range_digest`` (the #723 S3 requirement: "contract": contract, "env_seal": env_seal.seal_digest(dict(meta.get(env_seal.SEAL_KEY) or {})), "toolchain": cell_key.facts_digest(dict(meta.get("toolchain") or {})), - "code_closure": cell_key.facts_digest( - dict(meta.get("code_closure") or {})), }) diff --git a/src/gen_worker/cell_key.py b/src/gen_worker/cell_key.py index dae2cf72..e4b3d560 100644 --- a/src/gen_worker/cell_key.py +++ b/src/gen_worker/cell_key.py @@ -35,14 +35,15 @@ + the bundled ptxas/nvdisasm binaries. Replaces the old torch/triton/cuda/diffusers/transformers VERSION axes — content, never version strings - code_closure CONTENT digest of the STATIC import-graph closure of the - compile/composition code (compile_cache. - static_code_closure): the source files that shape the - traced graphs, found by pure static analysis from the - compile entrypoints. Sound because of the root-imports - convention (no runtime imports); the mint-time - completeness gate makes that convention a hard check. - Replaces the old gen_worker version axis + +ck6 (pgw#990) DROPS ``code_closure`` from the key. It is still RECORDED on +every artifact and still drives ``compile_cache``'s local re-trace memo, but +it is not identity: Paul's final ruling is that identity is the COMPUTATION +(traced graph x sm x toolchain x env_seal) and "code hashes are a memo, never +identity". A 147-file content hash made every wheel release re-key every cell +in the fleet for edits that could not change a traced graph — 0.93.0 -> 0.93.1 +moved the sdxl key on three plumbing files. Cells are stranded by SCHEME here, +by name, once, instead of silently on every release. Axes deliberately NOT in the key, recorded in metadata for observability and runtime compat checks (``compile_cache.verify``) only: ``sku`` (pgw#691 @@ -72,7 +73,7 @@ # digest — toolchain + code_closure content joined; every version axis # left. ck5 is FINAL: new identity facts ride the content digests (seal_v / # closure/toolchain values), never new axes. -KEY_SCHEME = "ck5" +KEY_SCHEME = "ck6" _PREFIX = KEY_SCHEME + "-" # The key digest doubles as the store flavor token, whose shared grammar # (th#597 C5: [a-z0-9][a-z0-9._-]{0,63}, Go+Py identical) caps tokens at 64 @@ -82,7 +83,7 @@ # Axes that must be non-empty for a computable key: a runtime that cannot # state them has no cell identity (CPU-only build, failed CUDA probe). _REQUIRED = ("format", "kind", "family", "sm", "contract", "env_seal", - "toolchain", "code_closure") + "toolchain") # Axes that may be legitimately absent ("" => omitted from canonical form): # lane "" is the plain-resident graph family; mode "" is whole-graph # compilation ("regional" per-block cells are different artifacts, ie#381). @@ -115,12 +116,31 @@ def digest(self) -> str: def is_key(value: str) -> bool: - """True when ``value`` is a cell-key digest string.""" + """True when ``value`` has cell-key SHAPE: ``ck`` + 1-2 scheme digits + + ``-`` + 56 lowercase hex. + + Scheme-AGNOSTIC, byte-for-byte the grammar tensorhub's + ``compilecache.IsCellKey`` enforces, and for the same reason it gives + (th#1183): pinning the current scheme here turns every other-scheme cell + into ``unreadable_cell_key``, which is both a lie and a filter no axis + justifies. A cell of an older scheme is admitted to the candidate list and + then ruled on by the axes that actually decide whether this runtime can + execute it — the artifact contract, the identity axes and the numerics + gate — not by the label on it. + """ v = str(value or "") + rest = v[2:] if v.startswith("ck") else "" + if not rest: + return False + digits = 0 + while digits < len(rest) and rest[digits].isdigit(): + digits += 1 + if not 1 <= digits <= 2 or digits >= len(rest) or rest[digits] != "-": + return False + hexpart = rest[digits + 1:] return ( - v.startswith(_PREFIX) - and len(v) == len(_PREFIX) + _DIGEST_HEX - and all(c in "0123456789abcdef" for c in v[len(_PREFIX):]) + len(hexpart) == _DIGEST_HEX + and all(c in "0123456789abcdef" for c in hexpart) ) @@ -209,8 +229,6 @@ def compute( "contract": str(contract or ""), "env_seal": env_seal.seal_digest(env_seal.effective_seal()), "toolchain": facts_digest(dict(cc.toolchain_digest())), - "code_closure": facts_digest( - dict(cc.static_code_closure(tuple(closure_roots)))), }) @@ -254,13 +272,9 @@ def from_artifact_metadata(meta: Mapping[str, Any]) -> CellKey: "execution environment is unproven" ) toolchain = meta.get("toolchain") - closure = meta.get("code_closure") - if not isinstance(toolchain, dict) or not toolchain \ - or not isinstance(closure, dict) or not closure: + if not isinstance(toolchain, dict) or not toolchain: raise CellKeyError( - "artifact records no toolchain/code_closure blocks (pre-ck5 " - "cell); no recipe identity" - ) + "artifact records no toolchain block; no recipe identity") return from_axes({ "format": str(meta.get("format") or ""), "kind": "inductor", @@ -274,7 +288,6 @@ def from_artifact_metadata(meta: Mapping[str, Any]) -> CellKey: "contract": contract_digest(contract_facts), "env_seal": env_seal.seal_digest(seal), "toolchain": facts_digest(toolchain), - "code_closure": facts_digest(closure), }) diff --git a/src/gen_worker/config/loader.py b/src/gen_worker/config/loader.py index 8362d0a8..8c7910aa 100644 --- a/src/gen_worker/config/loader.py +++ b/src/gen_worker/config/loader.py @@ -50,7 +50,6 @@ "TENSORHUB_CAS_DIR": "tensorhub_cas_dir", "TENSORHUB_FILL_SOURCE_DIR": "tensorhub_fill_source_dir", "CIVITAI_API_KEY": "civitai_api_key", - "GEN_WORKER_PREFER_AOT": "compile_prefer_aot", "GEN_WORKER_C2PA_CERT_PEM": "c2pa_cert_pem", "GEN_WORKER_C2PA_CERT_PATH": "c2pa_cert_path", # th#1307: GEN_WORKER_C2PA_KEY_PEM / _KEY_PATH are deliberately NOT diff --git a/src/gen_worker/config/settings.py b/src/gen_worker/config/settings.py index b6b948e4..761f6acc 100644 --- a/src/gen_worker/config/settings.py +++ b/src/gen_worker/config/settings.py @@ -151,18 +151,6 @@ class Settings(msgspec.Struct, frozen=True, kw_only=True): # Civitai provider credential (CIVITAI_API_KEY, alias CIVITAI_TOKEN). civitai_api_key: str = "" - # pgw#722 pilot flip switch (SDXL-AOT-PILOT-RUNBOOK.md §3): serving pods - # ALSO discover published aot-inductor cells (fetch-and-filter at arm), - # install the lifted-LoRA binding at adoption, and route adapter attach - # through the binding views. Default OFF = published AOT cells stay dark - # and behavior is byte-identical to 0.76.x. This is a typed pod-launch - # knob (release env -> Settings), NOT an env_seal knob: it selects which - # ARTIFACT to arm and changes no traced graph, so it must never join the - # sealed config table — a seal change would re-digest the env_seal axis - # and strand every published JIT cell fleet-wide, breaking the runbook's - # "flag off = boot back onto the dynamo cell" rollback. - compile_prefer_aot: bool = False # GEN_WORKER_PREFER_AOT - # C2PA Content Credentials signing (th#714, EU AI Act Art. 50). # Signing is ON iff cert material is set (inline PEM or path): every # generated media asset gets a signed provenance manifest at save time diff --git a/src/gen_worker/fleet_cells.py b/src/gen_worker/fleet_cells.py index 8a07a4b1..b3e77fa7 100644 --- a/src/gen_worker/fleet_cells.py +++ b/src/gen_worker/fleet_cells.py @@ -1025,15 +1025,21 @@ def _arming_policy( elif not delegate: delegate_refusal = "caller_forced_in_process" - # pgw#722 F1 (flag-gated, default OFF): PREFER a published aot-inductor - # cell over the delivered dynamo artifact. Discovery is fetch-and-filter - # (the worker cannot compute a stamped AOT key); the downloaded artifact - # rides the SAME choke point below, so the pgw#709 receipt gate and the - # aot_serve arm gates run unchanged. Any miss/failure falls through to - # today's policy with the originally delivered artifact. + # PREFER a published aot-inductor cell over the delivered dynamo artifact. + # Discovery is fetch-and-filter (the worker cannot compute a stamped AOT + # key); the downloaded artifact rides the SAME choke point below, so the + # pgw#709 receipt gate and the aot_serve arm gates run unchanged. Any + # miss/failure falls through to today's policy with the originally + # delivered artifact. + # + # pgw#990: UNCONDITIONAL. This was `aot_cells.prefer_aot()` — a default-OFF + # pilot switch carried by a release-scoped env entry — and the release + # stopped declaring the name on 2026-08-05, so every serving pod between + # then and attempt twenty-four skipped discovery entirely and self-minted + # over a published cell. Adoption is the path; a gate that un-arms it on a + # rebuild protects nothing. if ( - aot_cells.prefer_aot() - and family + family and publisher is not None and publisher.enabled() and cc.has_compile_target(pipe, cfg) @@ -1875,16 +1881,14 @@ def mint_recipe( The AOT lane was a pure CONSUMER: ``aot_cells.discover`` filtered for ``kind == "aot-inductor"`` artifacts and a miss fell through to the dynamo - self-mint, whose cell can never satisfy that filter. So a fleet with - ``prefer_aot`` armed missed, re-minted the wrong kind (or nothing), and - missed identically on every subsequent pod, forever. + self-mint, whose cell can never satisfy that filter. So a fleet missed, + re-minted the wrong kind (or nothing), and missed identically on every + subsequent pod, forever. Every decline here is NAMED on the wire. A silent decline is the defect class this issue exists to kill: five real L4 pods produced no mint and no refusal, which is indistinguishable from a crash. """ - if not aot_cells.prefer_aot(): - return RECIPE_DYNAMO family = str(getattr(cfg, "family", "") or "") def _decline(reason: str, detail: str) -> str: @@ -1892,7 +1896,7 @@ def _decline(reason: str, detail: str) -> str: if emit: activity_mod.emit_event( "self_mint_skipped", - f"family={family}: prefer_aot is armed but this miss cannot " + f"family={family}: this miss cannot " f"mint an aot-inductor cell — {detail}; falling back to the " f"dynamo self-mint (its artifact will NOT satisfy AOT " f"discovery, so a later pod misses again)", diff --git a/src/gen_worker/models/provision.py b/src/gen_worker/models/provision.py index 94ce9740..bbc4622f 100644 --- a/src/gen_worker/models/provision.py +++ b/src/gen_worker/models/provision.py @@ -286,7 +286,7 @@ def arm_aot( return AdoptOutcome.miss( "no_arm_for_mode", f"artifact declares mode={mode!r}, which this runtime has no arm for") - if bucket and current_or(_STANDALONE).compile_prefer_aot: + if bucket: from . import lora_lifted # The target module comes from the ARTIFACT's own recorded facts diff --git a/tests/test_adoption_key_pgw686.py b/tests/test_adoption_key_pgw686.py index 8fded5e3..8f0608a4 100644 --- a/tests/test_adoption_key_pgw686.py +++ b/tests/test_adoption_key_pgw686.py @@ -161,10 +161,15 @@ def test_burst_divergence_reproduced_execution_lane_only(burst_runtime: None) -> assert _requested("") != published assert _requested("fp8-hooks") != published assert _requested("") != _requested("fp8-hooks") - # pgw#691: the recorded ck2 burst keys are dead post-bump — is_key - # refuses them outright, so an old cell can only MISS, never half-match. + # pgw#691: the recorded ck2 burst keys are dead post-bump — they can + # never collide with a current key, so an old cell can only MISS, never + # half-match. pgw#990: they stay key-SHAPED (is_key mirrors tensorhub's + # scheme-agnostic IsCellKey, th#1183), and it is the AXES that refuse + # them, not the label. for old in (CK2_PUBLISHED, CK2_REQUESTED_PLAIN, CK2_REQUESTED_FP8_HOOKS): - assert not ck.is_key(old) + assert ck.is_key(old) + assert old != published + assert ck.mismatch(_BURST_META, old) != "" # --- the fix: one base-lane resolution for every cell-identity surface ----- diff --git a/tests/test_aot_adopt_events_pgw733.py b/tests/test_aot_adopt_events_pgw733.py index c8d7daed..1ea964d7 100644 --- a/tests/test_aot_adopt_events_pgw733.py +++ b/tests/test_aot_adopt_events_pgw733.py @@ -291,7 +291,6 @@ def _f1(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Any: from gen_worker import compile_cache as cc from gen_worker import config as gw_config - monkeypatch.setenv("GEN_WORKER_PREFER_AOT", "1") gw_config.reload_for_test() monkeypatch.setattr(cc, "has_compile_target", lambda pipe, cfg: True) art = tmp_path / "cell.tar.gz" diff --git a/tests/test_aot_flip_pgw722.py b/tests/test_aot_flip_pgw722.py index 3bc9e903..ad144aa0 100644 --- a/tests/test_aot_flip_pgw722.py +++ b/tests/test_aot_flip_pgw722.py @@ -1,9 +1,11 @@ -"""pgw#722 pilot flip seams (SDXL-AOT-PILOT-RUNBOOK.md §3) — F1/F2/F3. +"""pgw#722 F1/F2/F3 seams — UNGATED since pgw#990. -One flag (``Settings.compile_prefer_aot`` / ``GEN_WORKER_PREFER_AOT``), -default OFF. The red half of every seam asserts flag-off behavior is -IDENTICAL to today's: no discovery call, no lifted install, buffer-copy -adapter attach. The green half asserts each seam's contract: +There is no longer a flip switch. ``Settings.compile_prefer_aot`` / +``GEN_WORKER_PREFER_AOT`` are DELETED: the release-scoped env plumbing that +carried them silently un-armed adoption on every release rebuild, and three +A1 attempts self-minted over a published cell because of it. Adoption is the +path. The red half of every seam now asserts the gate cannot come back; the +green half asserts each seam's contract: * **F1** — fetch-and-filter cell discovery: list the family repo's checkpoints, keep only receipt-shaped ``aot-inductor`` cells this @@ -82,36 +84,32 @@ def _pilot_meta(key: str, *, execution_lane: str = "w8a8", bucket: int = 64, def _flag_on(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("GEN_WORKER_PREFER_AOT", "1") - gw_config.reload_for_test() + """pgw#990: there is nothing left to arm. Kept as a no-op so the green + seams below still read as "with adoption available" instead of being + rewritten into silence.""" # --------------------------------------------------------------------------- -# The flag +# pgw#990 — the gate is gone, and must not come back # --------------------------------------------------------------------------- -def test_flag_defaults_off() -> None: - assert gw_config.current().compile_prefer_aot is False - assert aot_cells.prefer_aot() is False - - -def test_flag_rides_typed_settings(monkeypatch: pytest.MonkeyPatch) -> None: - _flag_on(monkeypatch) - assert gw_config.current().compile_prefer_aot is True - assert aot_cells.prefer_aot() is True - +def test_no_prefer_aot_gate_survives_anywhere() -> None: + """The whole hardcut, asserted at every surface it lived on. -def test_flag_never_joins_the_env_seal(monkeypatch: pytest.MonkeyPatch) -> None: - """The flip switch selects an artifact; it must not change cell identity. - Flipping it must leave the env_seal digest byte-identical, or every - published JIT cell strands fleet-wide (the runbook's rollback contract).""" + It was not one flag: it was a Settings field, an env name, a loader + mapping, a module-level predicate and two call sites — and the release + that had to DECLARE the env name stopped declaring it, so adoption was + off fleet-wide from 2026-08-05 with nothing reporting it. + """ from gen_worker import env_seal + from gen_worker.config import loader + assert not hasattr(gw_config.Settings(), "compile_prefer_aot") + assert not hasattr(aot_cells, "prefer_aot") + assert "prefer_aot" not in aot_cells.__all__ + assert "GEN_WORKER_PREFER_AOT" not in loader._ENV_TO_FIELD assert "compile_prefer_aot" not in env_seal.CANONICAL_CONFIG - before = env_seal.seal_digest(env_seal.effective_seal()) - _flag_on(monkeypatch) - assert env_seal.seal_digest(env_seal.effective_seal()) == before # --------------------------------------------------------------------------- @@ -543,20 +541,32 @@ def _armable(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(cc, "has_compile_target", lambda pipe, cfg: True) -def test_flag_off_never_discovers( +def test_discovery_runs_with_no_flag_set( monkeypatch: pytest.MonkeyPatch, _armable: None, ) -> None: - """RED: flag off => byte-identical arming — discovery must not run.""" - def _forbidden(*args: Any, **kwargs: Any) -> None: - raise AssertionError("discovery ran with the flag off") + """pgw#990 RED: a pod with NO adoption env at all must still ask. - monkeypatch.setattr(aot_cells, "discover", _forbidden) + The inverse of the test this replaces. Every serving pod between + 2026-08-05 and attempt twenty-four passed the old assertion and was + thereby broken: it never called discover, self-minted for 75 minutes, and + published nothing anyone could see was wrong. + """ + monkeypatch.delenv("GEN_WORKER_PREFER_AOT", raising=False) + gw_config.reload_for_test() + asked: List[str] = [] + + def _discover(pipe: Any, cfg: Any, **kwargs: Any) -> None: + asked.append(str(getattr(cfg, "family", ""))) + return None + + monkeypatch.setattr(aot_cells, "discover", _discover) monkeypatch.setattr( fleet_cells.provision, "enable_compiled", lambda pipe, cfg, cache_dir, artifact: AdoptOutcome.hit()) outcome = fleet_cells.enable_compiled( _FakePipe(), _Cfg(), publisher=_StubPublisher()) # type: ignore[arg-type] - assert outcome.armed and outcome.self_mint is None + assert asked == [FAMILY] + assert outcome.armed def test_flag_on_arms_discovered_cell_and_advertises_it( @@ -697,15 +707,18 @@ def _enable(p: Any, c: Any, cache_dir: Any, artifact: Any) -> Any: return observed -def test_f2_flag_off_never_installs_lifting( +def test_f2_installs_lifting_with_no_flag_set( monkeypatch: pytest.MonkeyPatch, ) -> None: - """RED: flag off => the module reaches aot_serve.enable UNLIFTED, - exactly as 0.76.x ships (a lifted artifact then refuses by name).""" + """pgw#990: the lifted install is decided by the artifact's own bucket, + never by a pod env. It used to be `bucket and compile_prefer_aot`, so an + adopted bucket-bearing cell reached aot_serve.enable UNLIFTED and refused + by name on exactly the pods the flag had not reached.""" + monkeypatch.delenv("GEN_WORKER_PREFER_AOT", raising=False) + gw_config.reload_for_test() pipe = _Pipe() observed = _arm_aot(monkeypatch, pipe, enable_result=False) - assert observed == [False] - assert lora_lifted.lifted_binding(pipe.unet) is None + assert observed == [True] def test_f2_flag_on_installs_before_enable_in_proven_order( @@ -814,3 +827,49 @@ def test_f3_lifted_attach_writes_through_the_binding_views( residency.deactivate("ref", pipe, request_id="off") a, b = binding.tensors assert torch.count_nonzero(b) == 0 + + +def test_every_dynamo_fallback_in_mint_recipe_emits_a_decline() -> None: + """pgw#990, generalized: a gate that returns a DIFFERENT recipe is a + decline, and a decline must be on the wire. + + The deleted `prefer_aot` gate was the only one of `mint_recipe`'s paths + that returned RECIPE_DYNAMO without calling `_decline`, so it emitted + nothing at all — the sole tell that a fleet was minting the wrong KIND was + the word "dynamo" inside a SUCCESS event's detail string. The z-image lane + reproduced exactly that on sm_80 while the sdxl lane reproduced it on + sm_89, neither with an event to find. + + Asserted structurally rather than by exercising nine branches: inside + `mint_recipe`, the ONLY `return RECIPE_DYNAMO` may be the one inside + `_decline`, which emits before it returns. A new silent gate then fails + here instead of on a rented pod. + """ + import ast + import inspect + import textwrap + + tree = ast.parse(textwrap.dedent(inspect.getsource(fleet_cells.mint_recipe))) + fn = tree.body[0] + assert isinstance(fn, ast.FunctionDef) + + emitting: set[int] = set() + for node in ast.walk(fn): + if isinstance(node, ast.FunctionDef) and node.name == "_decline": + emitting = {id(n) for n in ast.walk(node)} + break + assert emitting, "mint_recipe no longer has the _decline helper" + + silent = [ + node.lineno + for node in ast.walk(fn) + if isinstance(node, ast.Return) + and isinstance(node.value, ast.Name) + and node.value.id == "RECIPE_DYNAMO" + and id(node) not in emitting + ] + assert not silent, ( + "mint_recipe returns RECIPE_DYNAMO without emitting a decline at " + f"line(s) {silent} — a gate that changes the recipe must say so on " + "the wire (pgw#990)" + ) diff --git a/tests/test_aot_lifted_arm_pgw822.py b/tests/test_aot_lifted_arm_pgw822.py index 57918b12..6480d314 100644 --- a/tests/test_aot_lifted_arm_pgw822.py +++ b/tests/test_aot_lifted_arm_pgw822.py @@ -289,7 +289,6 @@ def test_the_parent_declines_the_mint_by_name_instead_of_renting(monkeypatch) -> family=FAMILY, lora_bucket=BUCKET, shapes=(), text_lens=(), guidance_scales=(), targets=("unet",)) - monkeypatch.setattr(fleet_cells.aot_cells, "prefer_aot", lambda: True) monkeypatch.setattr(aot_mint, "lifted_torch_gap", lambda spec: "") events: list = [] monkeypatch.setattr( diff --git a/tests/test_aot_mint_unblock_pgw813_pgw815.py b/tests/test_aot_mint_unblock_pgw813_pgw815.py index cf0a8380..a57df76f 100644 --- a/tests/test_aot_mint_unblock_pgw813_pgw815.py +++ b/tests/test_aot_mint_unblock_pgw813_pgw815.py @@ -143,7 +143,7 @@ def _phases(events: List[Tuple[str, str, str]], kind: str) -> List[str]: @pytest.fixture() def _w8a8_miss(monkeypatch: pytest.MonkeyPatch) -> Any: - """The measured pod, reduced: prefer_aot armed, a real w8a8 lane, a + """The measured pod, reduced: a real w8a8 lane, a resolvable compile target, CUDA + toolchain present, and no cell. `mandatory_serving` is deliberately NOT stubbed — the w8a8 weight-lane @@ -152,7 +152,6 @@ def _w8a8_miss(monkeypatch: pytest.MonkeyPatch) -> Any: """ from gen_worker import aot_cells - monkeypatch.setenv("GEN_WORKER_PREFER_AOT", "1") monkeypatch.delenv("GEN_WORKER_MINT_IN_PROCESS", raising=False) monkeypatch.delenv("GEN_WORKER_EAGER_FIRST_BOOT", raising=False) gw_config.reload_for_test() @@ -253,7 +252,7 @@ def test_a_w8a8_miss_mints_AOT_and_not_dynamo( pending = outcome.self_mint assert pending is not None, "the miss produced no mint at all" assert pending.recipe == fleet_cells.RECIPE_AOT, ( - "a w8a8 miss with prefer_aot armed and a declaration registered must " + "a w8a8 miss with a declaration registered must " "mint an AOT cell; a dynamo artifact will never satisfy AOT discovery") assert pending.delegated is True assert "aot_requires_delegation" not in _phases(_events, "self_mint_skipped") diff --git a/tests/test_aot_selfmint_pgw805.py b/tests/test_aot_selfmint_pgw805.py index be65ff06..1ef58ebf 100644 --- a/tests/test_aot_selfmint_pgw805.py +++ b/tests/test_aot_selfmint_pgw805.py @@ -1,7 +1,7 @@ """pgw#805 — an AOT cell-discovery MISS must start a mint, or refuse BY NAME. The defect this pins, measured on five real 0.78.0 L4 pods (hub-dispatched, -`GEN_WORKER_PREFER_AOT` armed release-scoped, `Compile(family="sdxl", +`Compile(family="sdxl", targets=("unet",))` declared, compile target advertised, discovery working through a 200 listing, `cell_mint_hold_granted` on every pod, trickle traffic earning pgw#677 background turns): @@ -120,7 +120,6 @@ def _miss(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: Everything the five measured pods had: the flag armed, a publisher, a resolvable compile target, CUDA and a toolchain — and no cell. """ - monkeypatch.setenv("GEN_WORKER_PREFER_AOT", "1") gw_config.reload_for_test() monkeypatch.setattr(aot_cells, "discover", lambda *a, **k: None) monkeypatch.setattr( @@ -167,7 +166,7 @@ def test_aot_discovery_miss_enqueues_an_aot_mint( _miss: None, _events: List[Tuple[str, str, str]], ) -> None: """RED at HEAD: the pending's recipe was always the dynamo capture, whose - artifact kind AOT discovery rejects — so a fleet with prefer_aot armed + artifact kind AOT discovery rejects — so a fleet could never produce the cell it was looking for.""" register_export_declaration(_declaration()) diff --git a/tests/test_cell_key.py b/tests/test_cell_key.py index 35a91965..be46a0d9 100644 --- a/tests/test_cell_key.py +++ b/tests/test_cell_key.py @@ -34,13 +34,13 @@ def __init__( _FACTS = cc.declared_contract_facts(_ContractCfg()) _CONTRACT = ck.contract_digest(_FACTS) -# ck5 recipe axes: version strings and image identity live in METADATA -# only (observability); content digests carry the identity. +# ck6 recipe axes: version strings and image identity live in METADATA only +# (observability); content digests carry the identity. pgw#990 dropped +# `code_closure` — a source-file content hash is a MEMO, never identity. _AXES = { "format": "2", "kind": "inductor", "family": "ltx-2.3", "lane": "w8a8", "sm": "sm_100", "contract": _CONTRACT, "env_seal": "aa00bb11cc22dd33", "toolchain": "bb11cc22dd33ee44", - "code_closure": "cc22dd33ee44ff55", } _RT = { @@ -66,7 +66,7 @@ def test_key_deterministic_and_axis_sensitive(): assert a.digest == ck.from_axes(dict(_AXES)).digest assert ck.is_key(a.digest) for axis in ("family", "lane", "sm", "contract", "env_seal", - "toolchain", "code_closure"): + "toolchain"): bumped = dict(_AXES, **{axis: _AXES[axis] + "x"}) assert ck.from_axes(bumped).digest != a.digest, axis @@ -112,15 +112,29 @@ def _meta(sku, sm="sm_86"): assert _meta("a40", sm="sm_89")["cell_key"] != a40["cell_key"] -def test_key_scheme_ck5_old_keys_never_half_match(): +def test_key_scheme_ck6_old_keys_never_half_match(): """Each axis-set change bumps the scheme (sku collapse -> ck3, env_seal - -> ck4, recipe identity -> ck5): an older digest is no longer a key at - all — a clean MISS, never a half-match.""" + -> ck4, recipe identity -> ck5, code_closure OUT -> ck6): an older digest + can never collide with a current one — a clean MISS, never a half-match. + + pgw#990: shape is scheme-AGNOSTIC, byte-identical to tensorhub's + `compilecache.IsCellKey`, for the reason th#1183 gives — pinning the + current scheme in the shape check turns every other-scheme cell into + `unreadable_cell_key`, which is a lie and a filter no axis justifies. An + old-scheme token IS key-shaped; it simply names no artifact this runtime + computes, and `mismatch` says so on the axes. + """ key = ck.from_axes(_AXES).digest - assert key.startswith("ck5-") - for dead in ("ck2-", "ck3-", "ck4-"): - assert not ck.is_key(dead + "a" * 56) - assert "not a cell key" in ck.mismatch({}, dead + "a" * 56) + assert key.startswith("ck6-") + for dead in ("ck2-", "ck3-", "ck4-", "ck5-"): + token = dead + "a" * 56 + assert ck.is_key(token), "an old-scheme token is still key-SHAPED" + assert token != key + # No artifact metadata => no computable key => a named miss, never ''. + assert ck.mismatch({}, token) != "" + assert not ck.is_key("ck-" + "a" * 56) # no scheme digits + assert not ck.is_key("ck6-" + "a" * 55) # wrong digest width + assert not ck.is_key("ck6-" + "A" * 56) # uppercase hex def test_compute_matches_artifact_metadata_stamp(fixed_runtime): diff --git a/tests/test_config_boundary_pgw931.py b/tests/test_config_boundary_pgw931.py index bc95cf90..a96088dc 100644 --- a/tests/test_config_boundary_pgw931.py +++ b/tests/test_config_boundary_pgw931.py @@ -150,11 +150,11 @@ def test_an_unknown_owned_env_is_reported(monkeypatch) -> None: def test_a_known_env_is_not_reported_as_unknown(monkeypatch) -> None: - monkeypatch.setenv("GEN_WORKER_PREFER_AOT", "1") + monkeypatch.setenv("GEN_WORKER_C2PA_CERT_PATH", "/tmp/cert.pem") monkeypatch.setenv("GEN_WORKER_COMPUTE_CHILD", "1") # IPC, deliberately unbound monkeypatch.setenv("GEN_WORKER_LOG_LEVEL", "DEBUG") # library knob, ditto reported = config.unrecognised_owned_env() - assert "GEN_WORKER_PREFER_AOT" not in reported + assert "GEN_WORKER_C2PA_CERT_PATH" not in reported assert "GEN_WORKER_COMPUTE_CHILD" not in reported assert "GEN_WORKER_LOG_LEVEL" not in reported @@ -198,3 +198,12 @@ def test_parent_control_installs_the_boot_credential(monkeypatch) -> None: assert worker_credential.current() == "boot-jwt-xyz", ( "building a control parent must install its boot credential — the " "child cannot sign through a parent that has none") + + +def test_the_retired_prefer_aot_env_is_reported_as_unknown(monkeypatch) -> None: + """pgw#990: the gate is deleted, so a pod still being handed the env name + must SAY SO. A stale endpoint_env row is the exact trap that hid the + un-armed adoption path for three A1 attempts; silence about it is how it + would hide again.""" + monkeypatch.setenv("GEN_WORKER_PREFER_AOT", "1") + assert "GEN_WORKER_PREFER_AOT" in config.unrecognised_owned_env() diff --git a/tests/test_cxx_toolchain_pgw823.py b/tests/test_cxx_toolchain_pgw823.py index 684372b6..4d08e9bb 100644 --- a/tests/test_cxx_toolchain_pgw823.py +++ b/tests/test_cxx_toolchain_pgw823.py @@ -99,7 +99,6 @@ def test_the_predicate_does_not_contradict_torch( def test_the_parent_declines_the_AOT_recipe_by_name( monkeypatch: pytest.MonkeyPatch, ) -> None: - monkeypatch.setattr(fleet_cells.aot_cells, "prefer_aot", lambda: True) monkeypatch.setattr(fleet_cells.cc, "cxx_toolchain_present", lambda: False) from gen_worker import aot_mint diff --git a/tests/test_declaration_cannot_break_serving_pgw853.py b/tests/test_declaration_cannot_break_serving_pgw853.py index 380679ae..97a0733c 100644 --- a/tests/test_declaration_cannot_break_serving_pgw853.py +++ b/tests/test_declaration_cannot_break_serving_pgw853.py @@ -242,7 +242,6 @@ def _events(monkeypatch: pytest.MonkeyPatch) -> List[Tuple[str, str, str]]: @pytest.fixture() def _miss(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Any: """A real AOT discovery miss on an otherwise mint-capable pod.""" - monkeypatch.setenv("GEN_WORKER_PREFER_AOT", "1") gw_config.reload_for_test() monkeypatch.setattr(aot_cells, "discover", lambda *a, **k: None) monkeypatch.setattr( diff --git a/tests/test_determinism_pgw694.py b/tests/test_determinism_pgw694.py index 35ed65ed..805016f0 100644 --- a/tests/test_determinism_pgw694.py +++ b/tests/test_determinism_pgw694.py @@ -240,8 +240,11 @@ def test_ck5_requires_and_recomputes_the_env_seal(monkeypatch: Any) -> None: facts = cc.declared_contract_facts(_cfg()) want = ck.compute("toyfam", contract=ck.contract_digest(facts)) - assert want.digest.startswith("ck5-") - assert not ck.is_key("ck3-" + "a" * 56) # older schemes are dead + assert want.digest.startswith("ck6-") + # pgw#990: an older scheme is key-SHAPED and simply names nothing this + # runtime computes — refused on axes, not on its label. + assert ck.is_key("ck3-" + "a" * 56) + assert want.digest != "ck3-" + "a" * 56 meta = cc.artifact_metadata( family="toyfam", shapes=((64, 64),), targets=("transformer",), diff --git a/tests/test_recipe_identity_ck5.py b/tests/test_recipe_identity_ck5.py index 2d80f674..7d3f2cf2 100644 --- a/tests/test_recipe_identity_ck5.py +++ b/tests/test_recipe_identity_ck5.py @@ -125,19 +125,22 @@ def test_ck5_axes_are_the_recipe(pinned_runtime: None, monkeypatch: pytest.MonkeyPatch) -> None: facts = cc.declared_contract_facts(_cfg()) key = ck.compute(FAMILY, contract=ck.contract_digest(facts)) - assert key.digest.startswith("ck5-") + assert key.digest.startswith("ck6-") axes = key.axes_dict() assert set(axes) == {"format", "kind", "family", "sm", "contract", - "env_seal", "toolchain", "code_closure"} + "env_seal", "toolchain"} # Version axes are GONE from the key: a version-string bump alone can # never re-key a cell (content digests decide). monkeypatch.setattr(cc, "gen_worker_version", lambda: "99.0.0") monkeypatch.setenv("WORKER_IMAGE_DIGEST", "sha256:other") assert ck.compute( FAMILY, contract=ck.contract_digest(facts)).digest == key.digest - # Older schemes are dead. - for dead in ("ck2-", "ck3-", "ck4-"): - assert not ck.is_key(dead + "a" * 56) + # Older schemes can never collide with a current key; they stay + # key-SHAPED (pgw#990 — is_key mirrors tensorhub's scheme-agnostic + # IsCellKey) and are ruled on by axes, not by their label. + for dead in ("ck2-", "ck3-", "ck4-", "ck5-"): + assert ck.is_key(dead + "a" * 56) + assert key.digest != dead + "a" * 56 # Version-string axes are rejected outright. with pytest.raises(ck.CellKeyError): ck.from_axes(dict(axes, torch="2.13.0")) @@ -167,17 +170,24 @@ def test_metadata_roundtrips_the_recipe_key(pinned_runtime: None) -> None: assert ck.mismatch(meta, want) == "" # The version strings ride metadata for observability only. assert meta["torch"] and meta["gen_worker"] - # A pre-ck5 cell (no recipe blocks) has no identity. - legacy = {k: v for k, v in meta.items() - if k not in ("toolchain", "code_closure")} + # A cell with no toolchain block has no recipe identity. + legacy = {k: v for k, v in meta.items() if k != "toolchain"} with pytest.raises(ck.CellKeyError, match="recipe"): ck.from_artifact_metadata(legacy) - # A recorded-closure drift names a different key, and mismatch names - # the axis. + # pgw#990: `code_closure` is still RECORDED and still drives the local + # re-trace memo, but it is NOT identity — drifting it must leave the key + # alone. It used to re-key every cell in the fleet whenever an unrelated + # plumbing file moved, which is what a 147-file content hash does. drifted = json.loads(json.dumps(meta)) + assert drifted["code_closure"], "the closure is still recorded" drifted["code_closure"]["gen_worker/compile_cache.py"] = "0" * 16 - assert ck.from_artifact_metadata(drifted).digest != want.digest - assert ck.mismatch(drifted, want).startswith("code_closure:") + assert ck.from_artifact_metadata(drifted).digest == want.digest + assert ck.mismatch(drifted, want) == "" + # A TOOLCHAIN drift is identity, and mismatch names the axis. + retooled = json.loads(json.dumps(meta)) + retooled["toolchain"]["torch"] = "0" * 16 + assert ck.from_artifact_metadata(retooled).digest != want.digest + assert ck.mismatch(retooled, want).startswith("toolchain:") def test_toolchain_covers_the_model_libraries() -> None: