pgw#990 + pgw#988: adoption stops being a pilot — the gate is deleted, the declare/verify contract is split, and code_closure leaves the key (0.93.2) - #512
Conversation
d1f4b50 to
86aa3ed
Compare
…sure leaves the key, and is_key stops pinning a scheme REBUILT on fdbd4b2. PR #513 landed pgw#988 independently while this branch was blocked on the signature rule, with materially the same design (verify_declared / verify_contract, wired into the staged path at aot_serve.py:922). That implementation is canonical; this branch drops its duplicate pgw#988 work rather than contest a merged sibling's, and their tests own that ground. What remains is pgw#990 only, and master still carries every line of it. THE DEFECT. `aot_cells.prefer_aot()` (`Settings.compile_prefer_aot` / `GEN_WORKER_PREFER_AOT`, default OFF) stood in front of TWO call sites: `fleet_cells.enable_compiled`'s discovery branch, and `fleet_cells.mint_recipe`'s first line, which returned RECIPE_DYNAMO whenever it was off. It reached pods through a release-scoped endpoint_env entry that had to both match the release (applies_to) AND be declared by it; neither held for any release after bdf54080. So the forge minted kind=torch-inductor-cache cells that AOT discovery rejects by name, and no serving pod ever looked for a cell. Last aot_cell_discovery event of any kind on the dev fleet: 2026-08-02. Three A1 attempts ran un-armed. The gate is DELETED, not re-pointed: Settings field, env name, loader mapping, predicate, both call sites. provision's lifted-LoRA install now keys off the artifact's own bucket (it was `bucket and compile_prefer_aot`, so an adopted bucket-bearing cell reached aot_serve.enable UNLIFTED on exactly the pods the flag had not reached). A pod still handed the retired env name reports it as unrecognised owned env. PROVEN ON HARDWARE 2026-08-06, pod zqwhoypxefadug (L40S, gen-worker 0.93.2): aot_cell_discovery/miss family=sdxl lane=w8a8-lora64 ... not_an_aot_cell=5 self_mint_started/aot family=sdxl recipe=aot key=ck6-f512ec0e... lane=w8a8-lora64 First recipe=aot this program has recorded, and the first discovery event since 2026-08-02 — including discovery correctly refusing the previous dynamo cell. code_closure LEAVES THE KEY; ck5 -> ck6. Identity is the COMPUTATION (traced graph x sm x toolchain x env_seal); code hashes are a memo. A 147-file content hash re-keyed every cell in the fleet for edits that cannot change a traced graph. Still recorded, still driving the local re-trace memo, no longer identity. is_key BECOMES SCHEME-AGNOSTIC, byte-identical to tensorhub's compilecache.IsCellKey and for the reason th#1183 gives. This is load-bearing for the ck6 bump: a ck5-pinned tree refuses every ck6 cell as `unreadable_cell_key` before consulting any axis, which would strand the cells 0.93.2 workers mint. Verified by running a 0.93.2-shaped declare through the current filter — which also confirmed the extra bounded summary fields 0.93.2 declares are tolerated by master's contract, so nothing strands on those either. A GATE THAT CHANGES THE RECIPE MUST SAY SO. The deleted flag was the only one of mint_recipe's paths that returned RECIPE_DYNAMO without calling _decline, so it emitted nothing; the sole tell was the word "dynamo" inside a SUCCESS event's detail. An AST guard now asserts the only `return RECIPE_DYNAMO` inside mint_recipe is the one in _decline, which emits before returning. RED-verified against v0.93.1. Gates on this tree: mypy clean (231 files), ruff clean, http-timeout and config-read guards pass, and 92 tests green across this lane's files PLUS the sibling's test_cell_declare_verify_split_pgw988.py. No version bump and no CHANGELOG edit: master bumps at a cut and lanes write changelog.d fragments. These changes already shipped in the 0.93.2 tag, cut from v0.93.1 so the A1 pod run had a pre-v1 wheel; this is how they reach master.
86aa3ed to
f687f3f
Compare
|
PAUL'S RULING 2026-08-07 (relayed; recorded at the top of tracker pgw#958): the ck6 move is wrong — the counters restart at v1 per DESIGN-RULINGS §1.27(g). "the ck6 is wrong and can be removed / deleted just fine. The v1 is the correct one." This PR's ck5→ck6 half must be dropped or rewritten to the v1 restart before landing; ck6-keyed cells already minted on the dev stack are deletable artifacts (purge + re-mint at v1). The pgw#988 declare/verify split this PR also carries has meanwhile landed on master as fdbd4b2 — rebase and verify parity per the earlier comment on #513. |
…everted DESIGN-RULINGS §1.27(g) — "every version counter in this codebase starts at v1". `cell_key.KEY_SCHEME` ck6 -> ck1, `env_seal.SEAL_VERSION` 6 -> 1, `guard_closure.MANIFEST_VERSION` 3 -> 1, in ONE commit with every test pin that moves with them: all three are digest inputs, so splitting them means three fleet-wide re-keys instead of one. THE COLLISION WITH pgw#990, AS PAUL RULED IT (2026-08-07): *"the ck6 is wrong and can be removed / deleted just fine. The v1 is the correct one."* PR #512 landed `ck5` -> `ck6` on master (88ee170) while this branch was held. The bump is reverted rather than carried forward, and the cells it keyed are the deletable artifacts the ruling says they are. pgw#990's OTHER TWO DECISIONS ARE UNTOUCHED — the ruling is about the counter's VALUE, not about them, and both are load-bearing: * `code_closure` stays OUT of the key. Identity is the COMPUTATION; a 147-file content hash re-keyed the fleet for edits that cannot change a traced graph. * `is_key` stays scheme-AGNOSTIC, byte-identical to tensorhub's `compilecache.IsCellKey`, for the reason th#1183 gives. Reverting it to a ck1-only check — which is what this branch carried before the rebase — would answer every foreign-scheme cell `unreadable_cell_key`, a refusal on a LABEL that no axis justifies. The three test files that conflicted are resolved that way: ck1 is what this runtime MINTS, and ck2..ck6 tokens stay key-SHAPED and are ruled on by axes. THE PURGE IS DONE, NOT ASSUMED. Re-issuing `ck1` is only honest once the corpus that already used it is gone — the corrected 2026-08-06 census found 71 ck1 cells on the master stack and 2 on dev, minted 2026-07-21..29, so the collision this issue was filed about was already on disk. §1.27(g) requires a persisted counter be reset BY MIGRATION with a stated cost, and the hub exposes no DELETE for `cell_store` at all, so the purge could never be a pgw action: it is tensorhub migration `20260806T064516Z-e68cfe98` (th#1636, merged `b424c1eb`), and it has APPLIED to both standing stacks — dev 2026-08-06 07:11:33Z, master 2026-08-06 20:34:11Z, verified in `public.migrations` rather than inferred from the merge. Post-purge census, read the same way: master cell_store/receipts/demand/revocations all 0; dev holds ONE ck5 row re-minted 2026-08-06 14:43 by a 0.93.1 pod, which under ck1 can only MISS (`cell_key.py:55-59` — a wrong key is never a refusal) and is left for the `cas-gc` follow-up. `cell_mint_obligations` is deliberately untouched: no `cell_key` column, no ambiguity. Also here, because they are the same "no stale numeral" sweep: the `aot_wrapper_split` telemetry label `version="v2"` becomes `lever="runimpl"` (it names which split fired and was never a version), and the ck5/ck6 numerals left in prose across `aot_cells`, `aot_serve`, `aot_package`, `aot_flatten`, `graph_hash`, `compile_cache` and the unreached-surface baseline are retired. `warm_spans.py` keeps its `ck5-a53e02a7…` verbatim — that is a quoted pod observation, not a live scheme reference. No ordering comparison anywhere reads any of the three; the only version compare in the tree is `aot_resume.BANK_V`, already an equality check at 1. `COZY_CELL_EPOCH` remains the designed disown-everything lever, so these counters never move for recall reasons again.
|
Coordination outcome, for the record — the ck6 half of this PR has been reverted on master by #504. This PR merged as
So #504 was rebased onto master and rewritten to take the counters to Nothing else of this PR is touched, and that is deliberate. The ruling is about the counter's value. Your other two decisions are load-bearing and are preserved verbatim in the resolution of all three conflicted test files:
The ck6-keyed cells this PR's hardware run minted ( No action needed on this PR — it is merged and stays merged. Filing this here so the ck5→ck6 line in its commit message is not read later as the current state. |
…everted (#504) DESIGN-RULINGS §1.27(g) — "every version counter in this codebase starts at v1". `cell_key.KEY_SCHEME` ck6 -> ck1, `env_seal.SEAL_VERSION` 6 -> 1, `guard_closure.MANIFEST_VERSION` 3 -> 1, in ONE commit with every test pin that moves with them: all three are digest inputs, so splitting them means three fleet-wide re-keys instead of one. THE COLLISION WITH pgw#990, AS PAUL RULED IT (2026-08-07): *"the ck6 is wrong and can be removed / deleted just fine. The v1 is the correct one."* PR #512 landed `ck5` -> `ck6` on master (88ee170) while this branch was held. The bump is reverted rather than carried forward, and the cells it keyed are the deletable artifacts the ruling says they are. pgw#990's OTHER TWO DECISIONS ARE UNTOUCHED — the ruling is about the counter's VALUE, not about them, and both are load-bearing: * `code_closure` stays OUT of the key. Identity is the COMPUTATION; a 147-file content hash re-keyed the fleet for edits that cannot change a traced graph. * `is_key` stays scheme-AGNOSTIC, byte-identical to tensorhub's `compilecache.IsCellKey`, for the reason th#1183 gives. Reverting it to a ck1-only check — which is what this branch carried before the rebase — would answer every foreign-scheme cell `unreadable_cell_key`, a refusal on a LABEL that no axis justifies. The three test files that conflicted are resolved that way: ck1 is what this runtime MINTS, and ck2..ck6 tokens stay key-SHAPED and are ruled on by axes. THE PURGE IS DONE, NOT ASSUMED. Re-issuing `ck1` is only honest once the corpus that already used it is gone — the corrected 2026-08-06 census found 71 ck1 cells on the master stack and 2 on dev, minted 2026-07-21..29, so the collision this issue was filed about was already on disk. §1.27(g) requires a persisted counter be reset BY MIGRATION with a stated cost, and the hub exposes no DELETE for `cell_store` at all, so the purge could never be a pgw action: it is tensorhub migration `20260806T064516Z-e68cfe98` (th#1636, merged `b424c1eb`), and it has APPLIED to both standing stacks — dev 2026-08-06 07:11:33Z, master 2026-08-06 20:34:11Z, verified in `public.migrations` rather than inferred from the merge. Post-purge census, read the same way: master cell_store/receipts/demand/revocations all 0; dev holds ONE ck5 row re-minted 2026-08-06 14:43 by a 0.93.1 pod, which under ck1 can only MISS (`cell_key.py:55-59` — a wrong key is never a refusal) and is left for the `cas-gc` follow-up. `cell_mint_obligations` is deliberately untouched: no `cell_key` column, no ambiguity. Also here, because they are the same "no stale numeral" sweep: the `aot_wrapper_split` telemetry label `version="v2"` becomes `lever="runimpl"` (it names which split fired and was never a version), and the ck5/ck6 numerals left in prose across `aot_cells`, `aot_serve`, `aot_package`, `aot_flatten`, `graph_hash`, `compile_cache` and the unreached-surface baseline are retired. `warm_spans.py` keeps its `ck5-a53e02a7…` verbatim — that is a quoted pod observation, not a live scheme reference. No ordering comparison anywhere reads any of the three; the only version compare in the tree is `aot_resume.BANK_V`, already an equality check at 1. `COZY_CELL_EPOCH` remains the designed disown-everything lever, so these counters never move for recall reasons again.
Closes the root cause of pgw#868 A1's ADOPT gate. Found entirely from the dev hub's own tables, $0.00, no pod.
pgw#990 — one un-armed env entry produced BOTH halves of the failure
aot_cells.prefer_aot()(Settings.compile_prefer_aot/GEN_WORKER_PREFER_AOT, the pgw#722 "pilot flip switch, default OFF") stood in front of two call sites, not one:fleet_cells.enable_compiledF1 branchaot_cells.discoverat allfleet_cells.mint_recipefirst lineif not prefer_aot(): return RECIPE_DYNAMO— the mint's KINDIt reached pods through a release-scoped
endpoint_enventry that had to both match the release (applies_to) and be declared by it. Neither held for any release afterbdf54080…(2026-08-02) —applies_tonames that one release id, andrelease_env_declarationshas rows for no release after it.Measured on the dev fleet:
aot_cell_discoveryevent of any kind: 2026-08-02 23:45:47Zself_mint_compile/load20:25:43Z →warmup_forward20:26:09Z. No discovery, ever.kind="torch-inductor-cache", noentries, nohost_isa— a dynamo envelope, which_candidatesrejects by name (not_an_aot_cell). Its mint event isjit_compilewith the dynamo phase table.So the forge minted the wrong kind and no pod looked for a cell. pgw#805's own docstring had described the loop verbatim, in advance: "a fleet missed, re-minted the wrong kind (or nothing), and missed identically on every subsequent pod, forever."
The gate is DELETED, not re-pointed — re-pointing
applies_toleaves the trap armed for the next release rebuild. Gone: the Settings field, the env name, the loader mapping, the predicate, both call sites.provision's lifted-LoRA install now keys off the artifact's own bucket (it wasbucket and compile_prefer_aot, so an adopted bucket-bearing cell reachedaot_serve.enableunlifted on exactly the pods the flag had not reached). A pod still handed the retired env name now reports it as unrecognised owned env, so a stale row is loud.pgw#988 — the declare and the filter were one contract with no single authority
th#1645 correctly dropped the unbounded
entriesmap from the control-plane declare (98% of a real cell's envelope; it put the declare over the hub's 32 MiB cap).aot_cellswent on verifying the full contract against that declare, so every cell published fromc2e52f5fon was rejected withmalformed declared contract: metadata declares no entries map.Two named levels now:
aot_serve.verify_declared— the PREFILTER. Rules on what a bounded declare can state (format, kind, weights-baked refusal, identity axes, host ISA, family) plus the new boundedentries_digest+entries_count, whichcontrol_plane_metadatafolds in at the same seam that strips the map, so the two halves are written in one place and cannot drift apart again.aot_serve.verify— the GATE, unchanged, and it now actually runs: over the staged artifact's own metadata inaot_serve.arm, raisingcontract_invalid. Previously the full verify ran ONLY against the declare and never against the artifact.The prefilter fails open on a silent summary and the gate fails closed on every entry. A prefilter that refuses whatever it does not recognise is precisely what pgw#988 was.
code_closureleaves the key; ck5 → ck6Paul's final cell-identity ruling: identity is the COMPUTATION (traced graph × sm × toolchain × env_seal); code hashes are a memo. The ck5
code_closureaxis was a 147-file source-content hash, so every wheel release re-keyed every cell in the fleet for edits that cannot change a traced graph — 0.93.0 → 0.93.1 moved the sdxl key on three plumbing files, and this PR's own gate deletion would have moved it again. Dropped from_REQUIRED, still recorded on every artifact, still driving the local re-trace memo.KEY_SCHEMEbumped so old cells strand by name, once, instead of silently re-keying under a reused prefix.cell_key.is_keyis now scheme-agnostic, byte-identical to tensorhub'scompilecache.IsCellKeyand for the reason th#1183 gives: pinning the current scheme turned every other-scheme cell intounreadable_cell_key, which is both a lie and a filter no axis justifies. No hub change needed —IsCellKeyvalidates shape only.Branch base
Cut from
v0.93.1, notmaster: master carries th#1597'scozy.scheduler.v1wire cut, and a worker built from it cannot talk to the current pre-v1 dev hub.v0.93.2is tagged from this branch head so the A1 pod run gets a pre-v1 wheel; this PR is how the same changes reach master, so the planned 0.94.0 does not reintroduce the un-armed gate, the broken verify and the closure-keyed ck5.Tests
tests/test_declare_verify_contract_pgw988.py(new, 8 rows) drives the realcontrol_plane_metadatainto the real_candidates— that seam is the whole point, since either side alone can be "correct" while the pair is broken. RED-verified against a pristinev0.93.1tree: 6 of 8 fail, headline row with the defect's exact wire signature{'verify:malformed declared contract: metadata declares no entries map': 1}.test_aot_flip_pgw722rewritten: the red half now asserts the gate cannot come back (field, module attr,__all__, loader map, env_seal) and that a pod with no adoption env still callsdiscover— the inverse of the test it replaces, which every broken pod passed.test_config_boundary_pgw931gains a row asserting a pod still handed the retired env name reports it.test_cell_key/test_recipe_identity_ck5updated for ck6 + scheme-agnostic shape; the closure-drift row is inverted (drifting a recorded closure must now leave the key alone) and a toolchain-drift row added in its place.Local:
mypyclean (231 files),ruffclean, all four lint guards pass, targeted suites green. Tracker: pgw#990 filed, pgw#868 A1 record corrected, th#1647 rewritten as disproven (hub half in tensorhub PR #880).