pgw#990 + pgw#988: adoption stops being a pilot — the gate is deleted, the declare/verify contract is split, and code_closure leaves the key (0.93.2)

changelog.d/pgw990.md

@@ -0,0 +1,56 @@
+- **pgw#990: AOT cell adoption was behind a default-OFF pilot flag, and the flag
+  stopped arriving on 2026-08-05 — so the fleet minted the WRONG KIND of cell and
+  never looked for one.** `aot_cells.prefer_aot()`
+  (`Settings.compile_prefer_aot` / `GEN_WORKER_PREFER_AOT`) stood in front of two
+  call sites, not one: `fleet_cells.enable_compiled`'s discovery branch, **and
+  `fleet_cells.mint_recipe`'s first line**, which returned `RECIPE_DYNAMO`
+  whenever it was off. The flag reached pods through a release-scoped
+  `endpoint_env` entry that had to both match the release (`applies_to`) **and**
+  be declared by it, and neither held for any release after `bdf54080…`.
+  Measured on the dev fleet: the last `aot_cell_discovery` event of any kind was
+  2026-08-02, three A1 attempts ran with adoption un-armed, and the one cell that
+  got published carried `kind="torch-inductor-cache"` — a dynamo envelope AOT
+  discovery rejects by name. pgw#805's docstring had described the loop verbatim
+  in advance: *"a fleet missed, re-minted the wrong kind (or nothing), and missed
+  identically on every subsequent pod, forever."* **The gate is DELETED**, not
+  re-pointed — the Settings field, the env name, the loader mapping, the
+  predicate and both call sites. `provision`'s lifted-LoRA install now keys off
+  the artifact's own bucket (it was `bucket and compile_prefer_aot`, so an
+  adopted bucket-bearing cell reached `aot_serve.enable` UNLIFTED on exactly the
+  pods the flag had not reached). A pod still handed the retired env name now
+  reports it as unrecognised owned env, so a stale row is loud rather than
+  silent.
+  **Proven on hardware** (pod `zqwhoypxefadug`, L40S, gen-worker 0.93.2):
+  `self_mint_started/aot … recipe=aot key=ck6-f512ec0e… lane=w8a8-lora64`, the
+  first `recipe=aot` this program has recorded, alongside the first
+  `aot_cell_discovery` event of any kind since 2026-08-02.
+- **pgw#990: `code_closure` leaves the cell key; `ck5` → `ck6`.** Identity is the
+  COMPUTATION — traced graph × sm × toolchain × env_seal — and code hashes are a
+  MEMO, never identity. The ck5 `code_closure` axis was a 147-file source-content
+  hash, so every wheel release re-keyed every cell in the fleet for edits that
+  cannot change a traced graph (0.93.0 → 0.93.1 moved the sdxl key on three
+  plumbing files). It is dropped from the required axes, still RECORDED on every
+  artifact, and still drives the local re-trace memo. `KEY_SCHEME` is bumped so
+  older cells strand by name, once, instead of silently re-keying under a reused
+  prefix.
+- **pgw#990: `cell_key.is_key` becomes scheme-AGNOSTIC** — `ck` + 1–2 scheme
+  digits + 56 hex, byte-identical to tensorhub's `compilecache.IsCellKey` and for
+  the reason th#1183 gives: pinning the current scheme turned every other-scheme
+  cell into `unreadable_cell_key`, which is both a lie and a filter no axis
+  justifies. **This is load-bearing for the ck6 bump above**: without it a worker
+  built from a ck5-pinned tree refuses every ck6 cell before consulting a single
+  axis, which would strand the cells minted by 0.93.2 workers. Verified by
+  running a 0.93.2-shaped declare through the current filter.
+- **pgw#990: a gate that changes the recipe must SAY SO.** The deleted flag was
+  the only one of `mint_recipe`'s paths that returned `RECIPE_DYNAMO` without
+  calling `_decline`, so it emitted nothing — the sole tell that a whole fleet was
+  minting the wrong kind was the word `dynamo` inside a SUCCESS event's detail.
+  An AST guard now asserts that the only `return RECIPE_DYNAMO` inside
+  `mint_recipe` is the one in `_decline`, which emits before it returns.
+
+  > **Release note.** These changes shipped in the **0.93.2** tag, which was cut
+  > from `v0.93.1` (pre-`cozy.scheduler.v1`) so the pgw#868 A1 pod run could use
+  > a wheel that talks to the pre-v1 dev hub. This is how they reach `master`.
+  > The 0.93.2 tag targets an UNSIGNED commit; that is recorded as a deliberate
+  > exception in pgw#990's tracker entry rather than fixed by rewriting released
+  > history.

docs/environment.md

@@ -63,7 +63,6 @@ this page covers the worker itself.
 | `TENSORHUB_URL` | `tensorhub_url` | standalone-CLI resolve base URL |
 | `TENSORHUB_CACHE_DIR` | `tensorhub_cache_dir` | THE cache/CAS root knob — move cache/CAS off `/tmp` (cozy local persistence). `cache_paths.tensorhub_cas_dir()` derives the CAS from this and nothing else, so this is also what isolates the standalone `cli/run.py` in tests |
 | `TENSORHUB_CAS_DIR` | `tensorhub_cas_dir` | narrow override consulted at only two sites (`models/provision.py`, `procsplit/parent.py`). `cache_paths.tensorhub_cas_dir()` IGNORES it — setting it does not move the CAS for most call sites, and does not isolate `cli/run.py` |
-| `GEN_WORKER_PREFER_AOT` | `compile_prefer_aot` | prefer AOT compile cells over JIT. Deliberately NOT part of `env_seal`'s sealed config table |
 | `TENSORHUB_FILL_SOURCE_DIR` | `tensorhub_fill_source_dir` | th#850 managed-tier ruling: an endpoint-scoped datacenter-warm CAS mount (RunPod volume), checked before R2 on a blob miss and write-through warmed from R2. Never the CAS root — that always stays `TENSORHUB_CACHE_DIR`/local. tensorhub sets this only when a volume is attached; ismount-guarded, so a plain directory never gets mistaken for it |
 
 ## C2PA Content Credentials (Settings fields, th#714)

src/gen_worker/aot_cells.py

@@ -1,14 +1,14 @@
-"""AOT cell discovery — the pgw#722 pilot's F1 seam (flag-gated, default OFF).
+"""AOT cell discovery — the F1 seam every serving pod runs (pgw#722, pgw#990).
 
 A serving pod cannot COMPUTE an exported cell's key: an ``aot-inductor``
 cell is STAMPED at mint with axes only the mint holds (its traced-graph
 contract), so the runtime's ``cell_key.compute`` (kind="inductor") can
 never name it and the hub's worker-owned pull-by-key delivery (th#883)
 never lights it up. Published AOT cells are therefore provably dark.
 
-This module is the minimal pilot delivery: **fetch-and-filter**. At arm
-time (``fleet_cells.enable_compiled``, behind ``Settings.
-compile_prefer_aot``) the worker lists the family cell repo's checkpoints
+This module is **fetch-and-filter**. At arm time
+(``fleet_cells.enable_compiled``, unconditionally since pgw#990) the worker
+lists the family cell repo's checkpoints
 through the hub's existing catalog read API, filters for a cell THIS
 runtime can serve, downloads the artifact, and feeds it into the existing
 ``provision.enable_compiled`` HIT path — where the pgw#709 receipt gate
@@ -57,8 +57,6 @@
 from . import boot_phases as boot_mod
 from . import compile_cache as cc
 from .procsplit import broker
-from . import config
-from .config import Settings
 from .models.chunk_cas import sha256_file
 from .models.chunk_cas import (
     CAS_CHUNK_SIZE_BYTES,
@@ -97,15 +95,6 @@
 })
 
 
-def prefer_aot(settings: Optional[Settings] = None) -> bool:
-    """The pilot flip switch (typed pod-launch knob, default OFF).
-
-    Takes the `Settings` when the caller has them; otherwise reads the ones the
-    process entry installed (§1.18 — never the environment).
-    """
-    return bool((settings or config.current()).compile_prefer_aot)
-
-
 @dataclass(frozen=True)
 class AdoptedAotCell:
     """Identity of one discovered, downloaded, runtime-matched AOT cell.
@@ -505,5 +494,4 @@ def _discover_inner(
     "AdoptedAotCell",
     "EVENT",
     "discover",
-    "prefer_aot",
 ]

src/gen_worker/aot_mint.py

@@ -2878,8 +2878,6 @@ class hash folds that entry's ``range_digest`` (the #723 S3 requirement:
         "contract": contract,
         "env_seal": env_seal.seal_digest(dict(meta.get(env_seal.SEAL_KEY) or {})),
         "toolchain": cell_key.facts_digest(dict(meta.get("toolchain") or {})),
-        "code_closure": cell_key.facts_digest(
-            dict(meta.get("code_closure") or {})),
     })
 
 

src/gen_worker/cell_key.py

@@ -35,14 +35,15 @@
                   + the bundled ptxas/nvdisasm binaries. Replaces the old
                   torch/triton/cuda/diffusers/transformers VERSION axes —
                   content, never version strings
-    code_closure  CONTENT digest of the STATIC import-graph closure of the
-                  compile/composition code (compile_cache.
-                  static_code_closure): the source files that shape the
-                  traced graphs, found by pure static analysis from the
-                  compile entrypoints. Sound because of the root-imports
-                  convention (no runtime imports); the mint-time
-                  completeness gate makes that convention a hard check.
-                  Replaces the old gen_worker version axis
+
+ck6 (pgw#990) DROPS ``code_closure`` from the key. It is still RECORDED on
+every artifact and still drives ``compile_cache``'s local re-trace memo, but
+it is not identity: Paul's final ruling is that identity is the COMPUTATION
+(traced graph x sm x toolchain x env_seal) and "code hashes are a memo, never
+identity". A 147-file content hash made every wheel release re-key every cell
+in the fleet for edits that could not change a traced graph — 0.93.0 -> 0.93.1
+moved the sdxl key on three plumbing files. Cells are stranded by SCHEME here,
+by name, once, instead of silently on every release.
 
 Axes deliberately NOT in the key, recorded in metadata for observability
 and runtime compat checks (``compile_cache.verify``) only: ``sku`` (pgw#691
@@ -72,7 +73,7 @@
 # digest — toolchain + code_closure content joined; every version axis
 # left. ck5 is FINAL: new identity facts ride the content digests (seal_v /
 # closure/toolchain values), never new axes.
-KEY_SCHEME = "ck5"
+KEY_SCHEME = "ck6"
 _PREFIX = KEY_SCHEME + "-"
 # The key digest doubles as the store flavor token, whose shared grammar
 # (th#597 C5: [a-z0-9][a-z0-9._-]{0,63}, Go+Py identical) caps tokens at 64
@@ -82,7 +83,7 @@
 # Axes that must be non-empty for a computable key: a runtime that cannot
 # state them has no cell identity (CPU-only build, failed CUDA probe).
 _REQUIRED = ("format", "kind", "family", "sm", "contract", "env_seal",
-             "toolchain", "code_closure")
+             "toolchain")
 # Axes that may be legitimately absent ("" => omitted from canonical form):
 # lane "" is the plain-resident graph family; mode "" is whole-graph
 # compilation ("regional" per-block cells are different artifacts, ie#381).
@@ -115,12 +116,31 @@ def digest(self) -> str:
 
 
 def is_key(value: str) -> bool:
-    """True when ``value`` is a cell-key digest string."""
+    """True when ``value`` has cell-key SHAPE: ``ck`` + 1-2 scheme digits +
+    ``-`` + 56 lowercase hex.
+
+    Scheme-AGNOSTIC, byte-for-byte the grammar tensorhub's
+    ``compilecache.IsCellKey`` enforces, and for the same reason it gives
+    (th#1183): pinning the current scheme here turns every other-scheme cell
+    into ``unreadable_cell_key``, which is both a lie and a filter no axis
+    justifies. A cell of an older scheme is admitted to the candidate list and
+    then ruled on by the axes that actually decide whether this runtime can
+    execute it — the artifact contract, the identity axes and the numerics
+    gate — not by the label on it.
+    """
     v = str(value or "")
+    rest = v[2:] if v.startswith("ck") else ""
+    if not rest:
+        return False
+    digits = 0
+    while digits < len(rest) and rest[digits].isdigit():
+        digits += 1
+    if not 1 <= digits <= 2 or digits >= len(rest) or rest[digits] != "-":
+        return False
+    hexpart = rest[digits + 1:]
     return (
-        v.startswith(_PREFIX)
-        and len(v) == len(_PREFIX) + _DIGEST_HEX
-        and all(c in "0123456789abcdef" for c in v[len(_PREFIX):])
+        len(hexpart) == _DIGEST_HEX
+        and all(c in "0123456789abcdef" for c in hexpart)
     )
 
 
@@ -209,8 +229,6 @@ def compute(
         "contract": str(contract or ""),
         "env_seal": env_seal.seal_digest(env_seal.effective_seal()),
         "toolchain": facts_digest(dict(cc.toolchain_digest())),
-        "code_closure": facts_digest(
-            dict(cc.static_code_closure(tuple(closure_roots)))),
     })
 
 
@@ -254,13 +272,9 @@ def from_artifact_metadata(meta: Mapping[str, Any]) -> CellKey:
             "execution environment is unproven"
         )
     toolchain = meta.get("toolchain")
-    closure = meta.get("code_closure")
-    if not isinstance(toolchain, dict) or not toolchain \
-            or not isinstance(closure, dict) or not closure:
+    if not isinstance(toolchain, dict) or not toolchain:
         raise CellKeyError(
-            "artifact records no toolchain/code_closure blocks (pre-ck5 "
-            "cell); no recipe identity"
-        )
+            "artifact records no toolchain block; no recipe identity")
     return from_axes({
         "format": str(meta.get("format") or ""),
         "kind": "inductor",
@@ -274,7 +288,6 @@ def from_artifact_metadata(meta: Mapping[str, Any]) -> CellKey:
         "contract": contract_digest(contract_facts),
         "env_seal": env_seal.seal_digest(seal),
         "toolchain": facts_digest(toolchain),
-        "code_closure": facts_digest(closure),
     })
 
 

src/gen_worker/config/loader.py

@@ -50,7 +50,6 @@
     "TENSORHUB_CAS_DIR": "tensorhub_cas_dir",
     "TENSORHUB_FILL_SOURCE_DIR": "tensorhub_fill_source_dir",
     "CIVITAI_API_KEY": "civitai_api_key",
-    "GEN_WORKER_PREFER_AOT": "compile_prefer_aot",
     "GEN_WORKER_C2PA_CERT_PEM": "c2pa_cert_pem",
     "GEN_WORKER_C2PA_CERT_PATH": "c2pa_cert_path",
     # th#1307: GEN_WORKER_C2PA_KEY_PEM / _KEY_PATH are deliberately NOT

src/gen_worker/config/settings.py

@@ -151,18 +151,6 @@ class Settings(msgspec.Struct, frozen=True, kw_only=True):
     # Civitai provider credential (CIVITAI_API_KEY, alias CIVITAI_TOKEN).
     civitai_api_key: str = ""
 
-    # pgw#722 pilot flip switch (SDXL-AOT-PILOT-RUNBOOK.md §3): serving pods
-    # ALSO discover published aot-inductor cells (fetch-and-filter at arm),
-    # install the lifted-LoRA binding at adoption, and route adapter attach
-    # through the binding views. Default OFF = published AOT cells stay dark
-    # and behavior is byte-identical to 0.76.x. This is a typed pod-launch
-    # knob (release env -> Settings), NOT an env_seal knob: it selects which
-    # ARTIFACT to arm and changes no traced graph, so it must never join the
-    # sealed config table — a seal change would re-digest the env_seal axis
-    # and strand every published JIT cell fleet-wide, breaking the runbook's
-    # "flag off = boot back onto the dynamo cell" rollback.
-    compile_prefer_aot: bool = False  # GEN_WORKER_PREFER_AOT
-
     # C2PA Content Credentials signing (th#714, EU AI Act Art. 50).
     # Signing is ON iff cert material is set (inline PEM or path): every
     # generated media asset gets a signed provenance manifest at save time

src/gen_worker/fleet_cells.py

@@ -1025,15 +1025,21 @@ def _arming_policy(
     elif not delegate:
         delegate_refusal = "caller_forced_in_process"
 
-    # pgw#722 F1 (flag-gated, default OFF): PREFER a published aot-inductor
-    # cell over the delivered dynamo artifact. Discovery is fetch-and-filter
-    # (the worker cannot compute a stamped AOT key); the downloaded artifact
-    # rides the SAME choke point below, so the pgw#709 receipt gate and the
-    # aot_serve arm gates run unchanged. Any miss/failure falls through to
-    # today's policy with the originally delivered artifact.
+    # PREFER a published aot-inductor cell over the delivered dynamo artifact.
+    # Discovery is fetch-and-filter (the worker cannot compute a stamped AOT
+    # key); the downloaded artifact rides the SAME choke point below, so the
+    # pgw#709 receipt gate and the aot_serve arm gates run unchanged. Any
+    # miss/failure falls through to today's policy with the originally
+    # delivered artifact.
+    #
+    # pgw#990: UNCONDITIONAL. This was `aot_cells.prefer_aot()` — a default-OFF
+    # pilot switch carried by a release-scoped env entry — and the release
+    # stopped declaring the name on 2026-08-05, so every serving pod between
+    # then and attempt twenty-four skipped discovery entirely and self-minted
+    # over a published cell. Adoption is the path; a gate that un-arms it on a
+    # rebuild protects nothing.
     if (
-        aot_cells.prefer_aot()
-        and family
+        family
         and publisher is not None
         and publisher.enabled()
         and cc.has_compile_target(pipe, cfg)
@@ -1875,24 +1881,22 @@ def mint_recipe(
 
     The AOT lane was a pure CONSUMER: ``aot_cells.discover`` filtered for
     ``kind == "aot-inductor"`` artifacts and a miss fell through to the dynamo
-    self-mint, whose cell can never satisfy that filter. So a fleet with
-    ``prefer_aot`` armed missed, re-minted the wrong kind (or nothing), and
-    missed identically on every subsequent pod, forever.
+    self-mint, whose cell can never satisfy that filter. So a fleet missed,
+    re-minted the wrong kind (or nothing), and missed identically on every
+    subsequent pod, forever.
 
     Every decline here is NAMED on the wire. A silent decline is the defect
     class this issue exists to kill: five real L4 pods produced no mint and no
     refusal, which is indistinguishable from a crash.
     """
-    if not aot_cells.prefer_aot():
-        return RECIPE_DYNAMO
     family = str(getattr(cfg, "family", "") or "")
 
     def _decline(reason: str, detail: str) -> str:
         logger.info("fleet-cells: AOT mint declined (%s): %s", reason, detail)
         if emit:
             activity_mod.emit_event(
                 "self_mint_skipped",
-                f"family={family}: prefer_aot is armed but this miss cannot "
+                f"family={family}: this miss cannot "
                 f"mint an aot-inductor cell — {detail}; falling back to the "
                 f"dynamo self-mint (its artifact will NOT satisfy AOT "
                 f"discovery, so a later pod misses again)",

src/gen_worker/models/provision.py

@@ -286,7 +286,7 @@ def arm_aot(
         return AdoptOutcome.miss(
             "no_arm_for_mode",
             f"artifact declares mode={mode!r}, which this runtime has no arm for")
-    if bucket and current_or(_STANDALONE).compile_prefer_aot:
+    if bucket:
         from . import lora_lifted
 
         # The target module comes from the ARTIFACT's own recorded facts

tests/test_adoption_key_pgw686.py

@@ -161,10 +161,15 @@ def test_burst_divergence_reproduced_execution_lane_only(burst_runtime: None) ->
     assert _requested("") != published
     assert _requested("fp8-hooks") != published
     assert _requested("") != _requested("fp8-hooks")
-    # pgw#691: the recorded ck2 burst keys are dead post-bump — is_key
-    # refuses them outright, so an old cell can only MISS, never half-match.
+    # pgw#691: the recorded ck2 burst keys are dead post-bump — they can
+    # never collide with a current key, so an old cell can only MISS, never
+    # half-match. pgw#990: they stay key-SHAPED (is_key mirrors tensorhub's
+    # scheme-agnostic IsCellKey, th#1183), and it is the AXES that refuse
+    # them, not the label.
     for old in (CK2_PUBLISHED, CK2_REQUESTED_PLAIN, CK2_REQUESTED_FP8_HOOKS):
-        assert not ck.is_key(old)
+        assert ck.is_key(old)
+        assert old != published
+        assert ck.mismatch(_BURST_META, old) != ""
 
 
 # --- the fix: one base-lane resolution for every cell-identity surface -----

tests/test_aot_adopt_events_pgw733.py

@@ -291,7 +291,6 @@ def _f1(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Any:
     from gen_worker import compile_cache as cc
     from gen_worker import config as gw_config
 
-    monkeypatch.setenv("GEN_WORKER_PREFER_AOT", "1")
     gw_config.reload_for_test()
     monkeypatch.setattr(cc, "has_compile_target", lambda pipe, cfg: True)
     art = tmp_path / "cell.tar.gz"