pgw#988 (P0): the publish declare and the discovery filter are ONE contract - #513
Merged
Conversation
P0 regression on master, found by the pgw#978 rig and not by the lane that
caused it. th#1645 shrank the cell publish declare to control-plane size by
stripping the four envelope blocks that grow with the model — `entries` among
them. That was right about SIZE and it fixed a real 413 that had refused every
AOT cell publish thirty-two times. But `aot_cells._candidates` verifies the hub
LISTING metadata, and a listing row's metadata IS the declare body, with the
FULL `aot_serve.verify` — entries map and all. So from `c2e52f5f` every AOT cell
published by any worker was rejected on every pod as
verify:malformed declared contract: metadata declares no entries map
and a pod that finds no cell mints its own. The fleet re-minted on every cold
boot, forever, with a typed reject class that named the cause exactly and a
symptom that presented as cost rather than as an error.
Restoring `entries` to the declare would re-create th#1645, so the fix is the
split the transport already implies. `aot_serve.verify_declared` rules on
`DECLARED_AXES` — format, kind, the code-only flag, sm/torch/cuda, host ISA,
family — everything a bounded control-plane declare carries, and therefore
everything discovery may refuse a cell for before it moves a byte.
`verify_contract` rules on the `entries` map, which is unbounded in the model,
rides INSIDE the artifact, and is verified by `stage_artifact` on the unpacked
`metadata.json` — where `aot_serve` has always read it from in order to serve.
`verify` is both, for callers holding a full envelope (`stage_artifact`).
Nothing is checked less than it was; one check moved to the side of the fetch
that actually has the data, and the pre-download filter still refuses every axis
a declare CAN carry, by its own class, with no bytes moved.
The defect class is the one pgw#985 names a layer up: two independent
computations of one fact — what a declare must carry — with no single authority,
so a change to one side could not see the other. There is an authority now.
`aot_cells.DECLARE_CONTRACT_KEYS` is every key the pre-download filter reads off
a declare, and `fleet_cells.assert_declare_contract` checks at import that
nothing `control_plane_metadata` strips appears in it. The next block that moves
fails every test run rather than a fleet.
`tests/test_cell_declare_verify_split_pgw988.py` is the rig's own seam as a CI
row, minus the card: a real `CellPublisher.publish` over a real socket into a
hub that stores the declare it actually received, then a real
`aot_cells.discover` reading that same body back off a real checkpoint listing
and pulling the real artifact bytes the publisher uploaded. Nothing between the
two halves is stubbed, because the defect lived exactly between them — each half
was correct alone. Every existing discovery test hands `_candidates` a full
envelope no publisher has ever put on the wire, which is why the whole suite
stayed green through the regression. RED on the unfixed tree: 7/7 fail with the
verbatim reject class above. GREEN here.
Gates: 3300 passed, 37 skipped, 1 xfailed (tests/); 21 passed (tests_v2/); mypy
clean over 231 files; ruff clean; lint_http_timeouts, lint_unreached_surface and
lint_config_reads all exit 0.
Contributor
Author
|
Coordination note: PR #512 (the 0.93.2 train) titles itself as including 'the declare/verify contract is split' — the same pgw#988 fix this PR carries. Whichever lands first: if #512 lands, verify content parity with this branch (tests/test_cell_declare_verify_split_pgw988.py must be present and green) then close this PR as superseded; if this lands first, #512 should rebase. Do not merge both blindly. Tracker record: pgw#988 section, commit f4b22cdd. |
PaulFidika
added a commit
that referenced
this pull request
Aug 7, 2026
…sure leaves the key, and is_key stops pinning a scheme REBUILT on fdbd4b2. PR #513 landed pgw#988 independently while this branch was blocked on the signature rule, with materially the same design (verify_declared / verify_contract, wired into the staged path at aot_serve.py:922). That implementation is canonical; this branch drops its duplicate pgw#988 work rather than contest a merged sibling's, and their tests own that ground. What remains is pgw#990 only, and master still carries every line of it. THE DEFECT. `aot_cells.prefer_aot()` (`Settings.compile_prefer_aot` / `GEN_WORKER_PREFER_AOT`, default OFF) stood in front of TWO call sites: `fleet_cells.enable_compiled`'s discovery branch, and `fleet_cells.mint_recipe`'s first line, which returned RECIPE_DYNAMO whenever it was off. It reached pods through a release-scoped endpoint_env entry that had to both match the release (applies_to) AND be declared by it; neither held for any release after bdf54080. So the forge minted kind=torch-inductor-cache cells that AOT discovery rejects by name, and no serving pod ever looked for a cell. Last aot_cell_discovery event of any kind on the dev fleet: 2026-08-02. Three A1 attempts ran un-armed. The gate is DELETED, not re-pointed: Settings field, env name, loader mapping, predicate, both call sites. provision's lifted-LoRA install now keys off the artifact's own bucket (it was `bucket and compile_prefer_aot`, so an adopted bucket-bearing cell reached aot_serve.enable UNLIFTED on exactly the pods the flag had not reached). A pod still handed the retired env name reports it as unrecognised owned env. PROVEN ON HARDWARE 2026-08-06, pod zqwhoypxefadug (L40S, gen-worker 0.93.2): aot_cell_discovery/miss family=sdxl lane=w8a8-lora64 ... not_an_aot_cell=5 self_mint_started/aot family=sdxl recipe=aot key=ck6-f512ec0e... lane=w8a8-lora64 First recipe=aot this program has recorded, and the first discovery event since 2026-08-02 — including discovery correctly refusing the previous dynamo cell. code_closure LEAVES THE KEY; ck5 -> ck6. Identity is the COMPUTATION (traced graph x sm x toolchain x env_seal); code hashes are a memo. A 147-file content hash re-keyed every cell in the fleet for edits that cannot change a traced graph. Still recorded, still driving the local re-trace memo, no longer identity. is_key BECOMES SCHEME-AGNOSTIC, byte-identical to tensorhub's compilecache.IsCellKey and for the reason th#1183 gives. This is load-bearing for the ck6 bump: a ck5-pinned tree refuses every ck6 cell as `unreadable_cell_key` before consulting any axis, which would strand the cells 0.93.2 workers mint. Verified by running a 0.93.2-shaped declare through the current filter — which also confirmed the extra bounded summary fields 0.93.2 declares are tolerated by master's contract, so nothing strands on those either. A GATE THAT CHANGES THE RECIPE MUST SAY SO. The deleted flag was the only one of mint_recipe's paths that returned RECIPE_DYNAMO without calling _decline, so it emitted nothing; the sole tell was the word "dynamo" inside a SUCCESS event's detail. An AST guard now asserts the only `return RECIPE_DYNAMO` inside mint_recipe is the one in _decline, which emits before returning. RED-verified against v0.93.1. Gates on this tree: mypy clean (231 files), ruff clean, http-timeout and config-read guards pass, and 92 tests green across this lane's files PLUS the sibling's test_cell_declare_verify_split_pgw988.py. No version bump and no CHANGELOG edit: master bumps at a cut and lanes write changelog.d fragments. These changes already shipped in the 0.93.2 tag, cut from v0.93.1 so the A1 pod run had a pre-v1 wheel; this is how they reach master.
PaulFidika
added a commit
that referenced
this pull request
Aug 7, 2026
…sure leaves the key, and is_key stops pinning a scheme (#512) REBUILT on fdbd4b2. PR #513 landed pgw#988 independently while this branch was blocked on the signature rule, with materially the same design (verify_declared / verify_contract, wired into the staged path at aot_serve.py:922). That implementation is canonical; this branch drops its duplicate pgw#988 work rather than contest a merged sibling's, and their tests own that ground. What remains is pgw#990 only, and master still carries every line of it. THE DEFECT. `aot_cells.prefer_aot()` (`Settings.compile_prefer_aot` / `GEN_WORKER_PREFER_AOT`, default OFF) stood in front of TWO call sites: `fleet_cells.enable_compiled`'s discovery branch, and `fleet_cells.mint_recipe`'s first line, which returned RECIPE_DYNAMO whenever it was off. It reached pods through a release-scoped endpoint_env entry that had to both match the release (applies_to) AND be declared by it; neither held for any release after bdf54080. So the forge minted kind=torch-inductor-cache cells that AOT discovery rejects by name, and no serving pod ever looked for a cell. Last aot_cell_discovery event of any kind on the dev fleet: 2026-08-02. Three A1 attempts ran un-armed. The gate is DELETED, not re-pointed: Settings field, env name, loader mapping, predicate, both call sites. provision's lifted-LoRA install now keys off the artifact's own bucket (it was `bucket and compile_prefer_aot`, so an adopted bucket-bearing cell reached aot_serve.enable UNLIFTED on exactly the pods the flag had not reached). A pod still handed the retired env name reports it as unrecognised owned env. PROVEN ON HARDWARE 2026-08-06, pod zqwhoypxefadug (L40S, gen-worker 0.93.2): aot_cell_discovery/miss family=sdxl lane=w8a8-lora64 ... not_an_aot_cell=5 self_mint_started/aot family=sdxl recipe=aot key=ck6-f512ec0e... lane=w8a8-lora64 First recipe=aot this program has recorded, and the first discovery event since 2026-08-02 — including discovery correctly refusing the previous dynamo cell. code_closure LEAVES THE KEY; ck5 -> ck6. Identity is the COMPUTATION (traced graph x sm x toolchain x env_seal); code hashes are a memo. A 147-file content hash re-keyed every cell in the fleet for edits that cannot change a traced graph. Still recorded, still driving the local re-trace memo, no longer identity. is_key BECOMES SCHEME-AGNOSTIC, byte-identical to tensorhub's compilecache.IsCellKey and for the reason th#1183 gives. This is load-bearing for the ck6 bump: a ck5-pinned tree refuses every ck6 cell as `unreadable_cell_key` before consulting any axis, which would strand the cells 0.93.2 workers mint. Verified by running a 0.93.2-shaped declare through the current filter — which also confirmed the extra bounded summary fields 0.93.2 declares are tolerated by master's contract, so nothing strands on those either. A GATE THAT CHANGES THE RECIPE MUST SAY SO. The deleted flag was the only one of mint_recipe's paths that returned RECIPE_DYNAMO without calling _decline, so it emitted nothing; the sole tell was the word "dynamo" inside a SUCCESS event's detail. An AST guard now asserts the only `return RECIPE_DYNAMO` inside mint_recipe is the one in _decline, which emits before returning. RED-verified against v0.93.1. Gates on this tree: mypy clean (231 files), ruff clean, http-timeout and config-read guards pass, and 92 tests green across this lane's files PLUS the sibling's test_cell_declare_verify_split_pgw988.py. No version bump and no CHANGELOG edit: master bumps at a cut and lanes write changelog.d fragments. These changes already shipped in the 0.93.2 tag, cut from v0.93.1 so the A1 pod run had a pre-v1 wheel; this is how they reach master.
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The regression
c2e52f5f(th#1645 + pgw#987, PR #507) addedentriestofleet_cells._UNBOUNDED_ENVELOPE_BLOCKS, socontrol_plane_metadatastrips itfrom the publish declare. That is correct about SIZE —
entriesis unbounded inthe model, the declare is a control-plane call, and the change fixed a real 413
that had refused every AOT cell publish thirty-two times across two cards.
But
aot_cells._candidatesverifies the hub listing metadata — which IS thedeclare body — with the FULL
aot_serve.verify, entries map and all. So fromc2e52f5fevery AOT cell published by any worker is rejected on every pod:A pod that finds no cell mints its own, so the fleet re-mints on every cold boot
and the failure presents as cost, not as an error. Found by the pgw#978 rig,
whose cycle (mint → publish → a second process adopting) is the only thing
in-tree that crosses the seam.
The fix — a split, not a revert
Restoring
entriesto the declare re-creates th#1645, so the check moves to theside of the fetch that has the data:
aot_serve.verify_declaredDECLARED_AXES: format, kind, code-only flag, sm/torch/cuda, host ISA, familyaot_cells._candidates, pre-downloadaot_serve.verify_contractentriesmap + per-entryrange_digest/class_hash+ ck6 combined hashstage_artifact, on the unpackedmetadata.jsonaot_serve.verifyNothing is checked less than before.
entriesrides inside the artifact andaot_servehas always read it from there to serve; it is now verified theretoo, before anything is
dlopen'd. The pre-download filter still refuses everyaxis a declare CAN carry, by its own class, with no bytes moved (asserted).
The authority the contract lacked
The cause is two independent computations of one fact — what a declare must
carry — in two modules, so a change to one side could not see the other (the
pgw#985 defect class, one layer up). Now:
aot_cells.DECLARE_CONTRACT_KEYS— every key the pre-download filter reads;fleet_cells.assert_declare_contract()— runs at import, raises when_UNBOUNDED_ENVELOPE_BLOCKSintersects it.The next block that moves fails every test run rather than a fleet.
Test evidence
tests/test_cell_declare_verify_split_pgw988.py— the rig's own seam as a CIrow, minus the card: a real
CellPublisher.publishover a real socket into ahub that stores the declare it actually received, then a real
aot_cells.discoverreading that same body back off a real checkpoint listingand pulling the real artifact bytes the publisher uploaded. Nothing between the
two halves is stubbed; only the two hardware probes (
runtime_key, the torchread inside
host_isa.stamp) are pinned, and the ISA machine/level stay thishost's real ones.
Every existing discovery test hands
_candidatesa full envelope no publisherhas ever put on the wire — which is why the whole suite stayed green through the
regression.
origin/mastersrc tree: 7/7 fail, with the verbatimreject class
verify:malformed declared contract: metadata declares no entries map.Also asserted: the declare still carries none of the four unbounded blocks and
stays under
CELL_DECLARE_MAX_BYTESon the same wire that now yields anadoption — the fix does not undo th#1645.
Gates (run locally; GitHub Actions is in a major outage)
pytest tests/ -n 4 --dist loadfile— 3300 passed, 37 skipped, 1 xfailedpytest tests_v2/ -n 4 --dist loadfile— 21 passedmypy src/gen_worker— clean, 231 filesruff check src/gen_worker tests/test_cell_declare_verify_split_pgw988.py— cleanscripts/lint_http_timeouts.py,scripts/lint_unreached_surface.py,scripts/lint_config_reads.py— all exit 0Release
No version bump in this PR. This needs to reach the fleet: unadoptable cells
cost a full compile per cold boot on every serving pod, so it should ride the
next release train (0.93.2, already queued for the attempt-24 gate hardcut and
ck5→ck6) rather than be published solo. Recorded in the tracker.
Closes pgw#988.