Skip to content

pgw#990 + pgw#988: adoption stops being a pilot — the gate is deleted, the declare/verify contract is split, and code_closure leaves the key (0.93.2) - #512

Merged
PaulFidika merged 1 commit into
masterfrom
pgw990-aot-adopt-hardcut
Aug 7, 2026
Merged

pgw#990 + pgw#988: adoption stops being a pilot — the gate is deleted, the declare/verify contract is split, and code_closure leaves the key (0.93.2)#512
PaulFidika merged 1 commit into
masterfrom
pgw990-aot-adopt-hardcut

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

Closes the root cause of pgw#868 A1's ADOPT gate. Found entirely from the dev hub's own tables, $0.00, no pod.

pgw#990 — one un-armed env entry produced BOTH halves of the failure

aot_cells.prefer_aot() (Settings.compile_prefer_aot / GEN_WORKER_PREFER_AOT, the pgw#722 "pilot flip switch, default OFF") stood in front of two call sites, not one:

site what it gated
fleet_cells.enable_compiled F1 branch whether a serving pod calls aot_cells.discover at all
fleet_cells.mint_recipe first line if not prefer_aot(): return RECIPE_DYNAMO — the mint's KIND

It reached pods through a release-scoped endpoint_env entry that had to both match the release (applies_to) and be declared by it. Neither held for any release after bdf54080… (2026-08-02) — applies_to names that one release id, and release_env_declarations has rows for no release after it.

Measured on the dev fleet:

  • last aot_cell_discovery event of any kind: 2026-08-02 23:45:47Z
  • attempt 23's adopt pod: self_mint_compile/load 20:25:43Z → warmup_forward 20:26:09Z. No discovery, ever.
  • the one cell that got published: kind="torch-inductor-cache", no entries, no host_isaa dynamo envelope, which _candidates rejects by name (not_an_aot_cell). Its mint event is jit_compile with the dynamo phase table.

So the forge minted the wrong kind and no pod looked for a cell. pgw#805's own docstring had described the loop verbatim, in advance: "a fleet missed, re-minted the wrong kind (or nothing), and missed identically on every subsequent pod, forever."

The gate is DELETED, not re-pointed — re-pointing applies_to leaves the trap armed for the next release rebuild. Gone: the Settings field, the env name, the loader mapping, the predicate, both call sites. provision's lifted-LoRA install now keys off the artifact's own bucket (it was bucket and compile_prefer_aot, so an adopted bucket-bearing cell reached aot_serve.enable unlifted on exactly the pods the flag had not reached). A pod still handed the retired env name now reports it as unrecognised owned env, so a stale row is loud.

pgw#988 — the declare and the filter were one contract with no single authority

th#1645 correctly dropped the unbounded entries map from the control-plane declare (98% of a real cell's envelope; it put the declare over the hub's 32 MiB cap). aot_cells went on verifying the full contract against that declare, so every cell published from c2e52f5f on was rejected with malformed declared contract: metadata declares no entries map.

Two named levels now:

  • aot_serve.verify_declared — the PREFILTER. Rules on what a bounded declare can state (format, kind, weights-baked refusal, identity axes, host ISA, family) plus the new bounded entries_digest + entries_count, which control_plane_metadata folds in at the same seam that strips the map, so the two halves are written in one place and cannot drift apart again.
  • aot_serve.verify — the GATE, unchanged, and it now actually runs: over the staged artifact's own metadata in aot_serve.arm, raising contract_invalid. Previously the full verify ran ONLY against the declare and never against the artifact.

The prefilter fails open on a silent summary and the gate fails closed on every entry. A prefilter that refuses whatever it does not recognise is precisely what pgw#988 was.

code_closure leaves the key; ck5 → ck6

Paul's final cell-identity ruling: identity is the COMPUTATION (traced graph × sm × toolchain × env_seal); code hashes are a memo. The ck5 code_closure axis was a 147-file source-content hash, so every wheel release re-keyed every cell in the fleet for edits that cannot change a traced graph — 0.93.0 → 0.93.1 moved the sdxl key on three plumbing files, and this PR's own gate deletion would have moved it again. Dropped from _REQUIRED, still recorded on every artifact, still driving the local re-trace memo. KEY_SCHEME bumped so old cells strand by name, once, instead of silently re-keying under a reused prefix.

cell_key.is_key is now scheme-agnostic, byte-identical to tensorhub's compilecache.IsCellKey and for the reason th#1183 gives: pinning the current scheme turned every other-scheme cell into unreadable_cell_key, which is both a lie and a filter no axis justifies. No hub change neededIsCellKey validates shape only.

Branch base

Cut from v0.93.1, not master: master carries th#1597's cozy.scheduler.v1 wire cut, and a worker built from it cannot talk to the current pre-v1 dev hub. v0.93.2 is tagged from this branch head so the A1 pod run gets a pre-v1 wheel; this PR is how the same changes reach master, so the planned 0.94.0 does not reintroduce the un-armed gate, the broken verify and the closure-keyed ck5.

Tests

  • tests/test_declare_verify_contract_pgw988.py (new, 8 rows) drives the real control_plane_metadata into the real _candidates — that seam is the whole point, since either side alone can be "correct" while the pair is broken. RED-verified against a pristine v0.93.1 tree: 6 of 8 fail, headline row with the defect's exact wire signature {'verify:malformed declared contract: metadata declares no entries map': 1}.
  • test_aot_flip_pgw722 rewritten: the red half now asserts the gate cannot come back (field, module attr, __all__, loader map, env_seal) and that a pod with no adoption env still calls discover — the inverse of the test it replaces, which every broken pod passed.
  • test_config_boundary_pgw931 gains a row asserting a pod still handed the retired env name reports it.
  • test_cell_key / test_recipe_identity_ck5 updated for ck6 + scheme-agnostic shape; the closure-drift row is inverted (drifting a recorded closure must now leave the key alone) and a toolchain-drift row added in its place.

Local: mypy clean (231 files), ruff clean, all four lint guards pass, targeted suites green. Tracker: pgw#990 filed, pgw#868 A1 record corrected, th#1647 rewritten as disproven (hub half in tensorhub PR #880).

…sure leaves the key, and is_key stops pinning a scheme

REBUILT on fdbd4b2. PR #513 landed pgw#988 independently while this branch was
blocked on the signature rule, with materially the same design (verify_declared /
verify_contract, wired into the staged path at aot_serve.py:922). That
implementation is canonical; this branch drops its duplicate pgw#988 work rather
than contest a merged sibling's, and their tests own that ground. What remains is
pgw#990 only, and master still carries every line of it.

THE DEFECT. `aot_cells.prefer_aot()` (`Settings.compile_prefer_aot` /
`GEN_WORKER_PREFER_AOT`, default OFF) stood in front of TWO call sites:
`fleet_cells.enable_compiled`'s discovery branch, and `fleet_cells.mint_recipe`'s
first line, which returned RECIPE_DYNAMO whenever it was off. It reached pods
through a release-scoped endpoint_env entry that had to both match the release
(applies_to) AND be declared by it; neither held for any release after bdf54080.
So the forge minted kind=torch-inductor-cache cells that AOT discovery rejects by
name, and no serving pod ever looked for a cell. Last aot_cell_discovery event of
any kind on the dev fleet: 2026-08-02. Three A1 attempts ran un-armed.

The gate is DELETED, not re-pointed: Settings field, env name, loader mapping,
predicate, both call sites. provision's lifted-LoRA install now keys off the
artifact's own bucket (it was `bucket and compile_prefer_aot`, so an adopted
bucket-bearing cell reached aot_serve.enable UNLIFTED on exactly the pods the flag
had not reached). A pod still handed the retired env name reports it as
unrecognised owned env.

PROVEN ON HARDWARE 2026-08-06, pod zqwhoypxefadug (L40S, gen-worker 0.93.2):
  aot_cell_discovery/miss  family=sdxl lane=w8a8-lora64 ... not_an_aot_cell=5
  self_mint_started/aot    family=sdxl recipe=aot key=ck6-f512ec0e... lane=w8a8-lora64
First recipe=aot this program has recorded, and the first discovery event since
2026-08-02 — including discovery correctly refusing the previous dynamo cell.

code_closure LEAVES THE KEY; ck5 -> ck6. Identity is the COMPUTATION (traced graph
x sm x toolchain x env_seal); code hashes are a memo. A 147-file content hash
re-keyed every cell in the fleet for edits that cannot change a traced graph.
Still recorded, still driving the local re-trace memo, no longer identity.

is_key BECOMES SCHEME-AGNOSTIC, byte-identical to tensorhub's
compilecache.IsCellKey and for the reason th#1183 gives. This is load-bearing for
the ck6 bump: a ck5-pinned tree refuses every ck6 cell as `unreadable_cell_key`
before consulting any axis, which would strand the cells 0.93.2 workers mint.
Verified by running a 0.93.2-shaped declare through the current filter — which
also confirmed the extra bounded summary fields 0.93.2 declares are tolerated by
master's contract, so nothing strands on those either.

A GATE THAT CHANGES THE RECIPE MUST SAY SO. The deleted flag was the only one of
mint_recipe's paths that returned RECIPE_DYNAMO without calling _decline, so it
emitted nothing; the sole tell was the word "dynamo" inside a SUCCESS event's
detail. An AST guard now asserts the only `return RECIPE_DYNAMO` inside
mint_recipe is the one in _decline, which emits before returning. RED-verified
against v0.93.1.

Gates on this tree: mypy clean (231 files), ruff clean, http-timeout and
config-read guards pass, and 92 tests green across this lane's files PLUS the
sibling's test_cell_declare_verify_split_pgw988.py.

No version bump and no CHANGELOG edit: master bumps at a cut and lanes write
changelog.d fragments. These changes already shipped in the 0.93.2 tag, cut from
v0.93.1 so the A1 pod run had a pre-v1 wheel; this is how they reach master.
@PaulFidika
PaulFidika force-pushed the pgw990-aot-adopt-hardcut branch from 86aa3ed to f687f3f Compare August 7, 2026 00:53
@PaulFidika
PaulFidika merged commit 88ee170 into master Aug 7, 2026
2 checks passed
@PaulFidika
PaulFidika deleted the pgw990-aot-adopt-hardcut branch August 7, 2026 02:01
@PaulFidika

Copy link
Copy Markdown
Contributor Author

PAUL'S RULING 2026-08-07 (relayed; recorded at the top of tracker pgw#958): the ck6 move is wrong — the counters restart at v1 per DESIGN-RULINGS §1.27(g). "the ck6 is wrong and can be removed / deleted just fine. The v1 is the correct one." This PR's ck5→ck6 half must be dropped or rewritten to the v1 restart before landing; ck6-keyed cells already minted on the dev stack are deletable artifacts (purge + re-mint at v1). The pgw#988 declare/verify split this PR also carries has meanwhile landed on master as fdbd4b2 — rebase and verify parity per the earlier comment on #513.

PaulFidika added a commit that referenced this pull request Aug 7, 2026
…everted

DESIGN-RULINGS §1.27(g) — "every version counter in this codebase starts at
v1". `cell_key.KEY_SCHEME` ck6 -> ck1, `env_seal.SEAL_VERSION` 6 -> 1,
`guard_closure.MANIFEST_VERSION` 3 -> 1, in ONE commit with every test pin that
moves with them: all three are digest inputs, so splitting them means three
fleet-wide re-keys instead of one.

THE COLLISION WITH pgw#990, AS PAUL RULED IT (2026-08-07): *"the ck6 is wrong
and can be removed / deleted just fine. The v1 is the correct one."* PR #512
landed `ck5` -> `ck6` on master (88ee170) while this branch was held. The bump
is reverted rather than carried forward, and the cells it keyed are the
deletable artifacts the ruling says they are.

pgw#990's OTHER TWO DECISIONS ARE UNTOUCHED — the ruling is about the counter's
VALUE, not about them, and both are load-bearing:

  * `code_closure` stays OUT of the key. Identity is the COMPUTATION; a
    147-file content hash re-keyed the fleet for edits that cannot change a
    traced graph.
  * `is_key` stays scheme-AGNOSTIC, byte-identical to tensorhub's
    `compilecache.IsCellKey`, for the reason th#1183 gives. Reverting it to
    a ck1-only check — which is what this branch carried before the rebase —
    would answer every foreign-scheme cell `unreadable_cell_key`, a refusal on
    a LABEL that no axis justifies. The three test files that conflicted are
    resolved that way: ck1 is what this runtime MINTS, and ck2..ck6 tokens stay
    key-SHAPED and are ruled on by axes.

THE PURGE IS DONE, NOT ASSUMED. Re-issuing `ck1` is only honest once the
corpus that already used it is gone — the corrected 2026-08-06 census found 71
ck1 cells on the master stack and 2 on dev, minted 2026-07-21..29, so the
collision this issue was filed about was already on disk. §1.27(g) requires a
persisted counter be reset BY MIGRATION with a stated cost, and the hub exposes
no DELETE for `cell_store` at all, so the purge could never be a pgw action:
it is tensorhub migration `20260806T064516Z-e68cfe98` (th#1636, merged
`b424c1eb`), and it has APPLIED to both standing stacks — dev 2026-08-06
07:11:33Z, master 2026-08-06 20:34:11Z, verified in `public.migrations` rather
than inferred from the merge. Post-purge census, read the same way: master
cell_store/receipts/demand/revocations all 0; dev holds ONE ck5 row re-minted
2026-08-06 14:43 by a 0.93.1 pod, which under ck1 can only MISS
(`cell_key.py:55-59` — a wrong key is never a refusal) and is left for the
`cas-gc` follow-up. `cell_mint_obligations` is deliberately untouched: no
`cell_key` column, no ambiguity.

Also here, because they are the same "no stale numeral" sweep: the
`aot_wrapper_split` telemetry label `version="v2"` becomes `lever="runimpl"`
(it names which split fired and was never a version), and the ck5/ck6 numerals
left in prose across `aot_cells`, `aot_serve`, `aot_package`, `aot_flatten`,
`graph_hash`, `compile_cache` and the unreached-surface baseline are retired.
`warm_spans.py` keeps its `ck5-a53e02a7…` verbatim — that is a quoted pod
observation, not a live scheme reference.

No ordering comparison anywhere reads any of the three; the only version
compare in the tree is `aot_resume.BANK_V`, already an equality check at 1.
`COZY_CELL_EPOCH` remains the designed disown-everything lever, so these
counters never move for recall reasons again.
@PaulFidika

Copy link
Copy Markdown
Contributor Author

Coordination outcome, for the record — the ck6 half of this PR has been reverted on master by #504.

This PR merged as 88ee1709 carrying KEY_SCHEME ck5ck6. Paul ruled on the pgw#958 collision the same day:

"The ck6 is wrong and can be removed / deleted just fine. The v1 is the correct one." — 2026-08-07

So #504 was rebased onto master and rewritten to take the counters to ck1 / seal v1 / manifest v1 per DESIGN-RULINGS §1.27(g), reverting the bump this PR made.

Nothing else of this PR is touched, and that is deliberate. The ruling is about the counter's value. Your other two decisions are load-bearing and are preserved verbatim in the resolution of all three conflicted test files:

  • code_closure stays out of the key — identity is the COMPUTATION, code hashes are a memo.
  • is_key stays scheme-AGNOSTIC, byte-identical to compilecache.IsCellKey, for the reason th#1183 gives. Your PR body called this "load-bearing for the ck6 bump"; it is equally load-bearing for the ck1 restart, and more so — reverting it to a scheme-pinned check (which is what pgw#958: the three cell-identity counters restart at v1, and ck6 is reverted #504 carried before the rebase) would answer every ck2..ck6 cell unreadable_cell_key, a refusal on a label that no axis justifies.

The ck6-keyed cells this PR's hardware run minted (ck6-f512ec0e… on pod zqwhoypxefadug) are deletable artifacts under the ruling. The corpus purge is tensorhub migration 20260806T064516Z-e68cfe98 (th#1636, b424c1eb), already applied to both standing stacks — dev 2026-08-06 07:11:33Z, master 20:34:11Z, verified in public.migrations. Post-purge the master stack holds zero cell rows; dev holds one ck5 row re-minted afterwards by a 0.93.1 pod, which under ck1 can only MISS and is left to cas-gc.

No action needed on this PR — it is merged and stays merged. Filing this here so the ck5→ck6 line in its commit message is not read later as the current state.

PaulFidika added a commit that referenced this pull request Aug 7, 2026
…everted (#504)

DESIGN-RULINGS §1.27(g) — "every version counter in this codebase starts at
v1". `cell_key.KEY_SCHEME` ck6 -> ck1, `env_seal.SEAL_VERSION` 6 -> 1,
`guard_closure.MANIFEST_VERSION` 3 -> 1, in ONE commit with every test pin that
moves with them: all three are digest inputs, so splitting them means three
fleet-wide re-keys instead of one.

THE COLLISION WITH pgw#990, AS PAUL RULED IT (2026-08-07): *"the ck6 is wrong
and can be removed / deleted just fine. The v1 is the correct one."* PR #512
landed `ck5` -> `ck6` on master (88ee170) while this branch was held. The bump
is reverted rather than carried forward, and the cells it keyed are the
deletable artifacts the ruling says they are.

pgw#990's OTHER TWO DECISIONS ARE UNTOUCHED — the ruling is about the counter's
VALUE, not about them, and both are load-bearing:

  * `code_closure` stays OUT of the key. Identity is the COMPUTATION; a
    147-file content hash re-keyed the fleet for edits that cannot change a
    traced graph.
  * `is_key` stays scheme-AGNOSTIC, byte-identical to tensorhub's
    `compilecache.IsCellKey`, for the reason th#1183 gives. Reverting it to
    a ck1-only check — which is what this branch carried before the rebase —
    would answer every foreign-scheme cell `unreadable_cell_key`, a refusal on
    a LABEL that no axis justifies. The three test files that conflicted are
    resolved that way: ck1 is what this runtime MINTS, and ck2..ck6 tokens stay
    key-SHAPED and are ruled on by axes.

THE PURGE IS DONE, NOT ASSUMED. Re-issuing `ck1` is only honest once the
corpus that already used it is gone — the corrected 2026-08-06 census found 71
ck1 cells on the master stack and 2 on dev, minted 2026-07-21..29, so the
collision this issue was filed about was already on disk. §1.27(g) requires a
persisted counter be reset BY MIGRATION with a stated cost, and the hub exposes
no DELETE for `cell_store` at all, so the purge could never be a pgw action:
it is tensorhub migration `20260806T064516Z-e68cfe98` (th#1636, merged
`b424c1eb`), and it has APPLIED to both standing stacks — dev 2026-08-06
07:11:33Z, master 2026-08-06 20:34:11Z, verified in `public.migrations` rather
than inferred from the merge. Post-purge census, read the same way: master
cell_store/receipts/demand/revocations all 0; dev holds ONE ck5 row re-minted
2026-08-06 14:43 by a 0.93.1 pod, which under ck1 can only MISS
(`cell_key.py:55-59` — a wrong key is never a refusal) and is left for the
`cas-gc` follow-up. `cell_mint_obligations` is deliberately untouched: no
`cell_key` column, no ambiguity.

Also here, because they are the same "no stale numeral" sweep: the
`aot_wrapper_split` telemetry label `version="v2"` becomes `lever="runimpl"`
(it names which split fired and was never a version), and the ck5/ck6 numerals
left in prose across `aot_cells`, `aot_serve`, `aot_package`, `aot_flatten`,
`graph_hash`, `compile_cache` and the unreached-surface baseline are retired.
`warm_spans.py` keeps its `ck5-a53e02a7…` verbatim — that is a quoted pod
observation, not a live scheme reference.

No ordering comparison anywhere reads any of the three; the only version
compare in the tree is `aot_resume.BANK_V`, already an equality check at 1.
`COZY_CELL_EPOCH` remains the designed disown-everything lever, so these
counters never move for recall reasons again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant