Skip to content

pgw#988 (P0): the publish declare and the discovery filter are ONE contract - #513

Merged
PaulFidika merged 1 commit into
masterfrom
988-declare-verify-split
Aug 7, 2026
Merged

pgw#988 (P0): the publish declare and the discovery filter are ONE contract#513
PaulFidika merged 1 commit into
masterfrom
988-declare-verify-split

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

The regression

c2e52f5f (th#1645 + pgw#987, PR #507) added entries to
fleet_cells._UNBOUNDED_ENVELOPE_BLOCKS, so control_plane_metadata strips it
from the publish declare. That is correct about SIZE — entries is unbounded in
the model, the declare is a control-plane call, and the change fixed a real 413
that had refused every AOT cell publish thirty-two times across two cards.

But aot_cells._candidates verifies the hub listing metadata — which IS the
declare body — with the FULL aot_serve.verify, entries map and all. So from
c2e52f5f every AOT cell published by any worker is rejected on every pod:

aot-cells: ck5-… filtered: malformed declared contract: metadata declares no entries map
aot_cell_discovery miss: no matching aot-inductor cell among 1 checkpoint(s)
    — rejected by class: verify:malformed declared contract: metadata declares no entries map=1

A pod that finds no cell mints its own, so the fleet re-mints on every cold boot
and the failure presents as cost, not as an error. Found by the pgw#978 rig,
whose cycle (mint → publish → a second process adopting) is the only thing
in-tree that crosses the seam.

The fix — a split, not a revert

Restoring entries to the declare re-creates th#1645, so the check moves to the
side of the fetch that has the data:

rules on called from
aot_serve.verify_declared DECLARED_AXES: format, kind, code-only flag, sm/torch/cuda, host ISA, family aot_cells._candidates, pre-download
aot_serve.verify_contract the entries map + per-entry range_digest/class_hash + ck6 combined hash stage_artifact, on the unpacked metadata.json
aot_serve.verify both callers holding a full envelope

Nothing is checked less than before. entries rides inside the artifact and
aot_serve has always read it from there to serve; it is now verified there
too, before anything is dlopen'd. The pre-download filter still refuses every
axis a declare CAN carry, by its own class, with no bytes moved (asserted).

The authority the contract lacked

The cause is two independent computations of one fact — what a declare must
carry — in two modules, so a change to one side could not see the other (the
pgw#985 defect class, one layer up). Now:

  • aot_cells.DECLARE_CONTRACT_KEYS — every key the pre-download filter reads;
  • fleet_cells.assert_declare_contract() — runs at import, raises when
    _UNBOUNDED_ENVELOPE_BLOCKS intersects it.

The next block that moves fails every test run rather than a fleet.

Test evidence

tests/test_cell_declare_verify_split_pgw988.py — the rig's own seam as a CI
row, minus the card: a real CellPublisher.publish over a real socket into a
hub that stores the declare it actually received, then a real
aot_cells.discover reading that same body back off a real checkpoint listing
and pulling the real artifact bytes the publisher uploaded. Nothing between the
two halves is stubbed; only the two hardware probes (runtime_key, the torch
read inside host_isa.stamp) are pinned, and the ISA machine/level stay this
host's real ones.

Every existing discovery test hands _candidates a full envelope no publisher
has ever put on the wire — which is why the whole suite stayed green through the
regression.

  • RED on a clean origin/master src tree: 7/7 fail, with the verbatim
    reject class verify:malformed declared contract: metadata declares no entries map.
  • GREEN here: 7/7 pass.

Also asserted: the declare still carries none of the four unbounded blocks and
stays under CELL_DECLARE_MAX_BYTES on the same wire that now yields an
adoption — the fix does not undo th#1645.

Gates (run locally; GitHub Actions is in a major outage)

  • pytest tests/ -n 4 --dist loadfile3300 passed, 37 skipped, 1 xfailed
  • pytest tests_v2/ -n 4 --dist loadfile21 passed
  • mypy src/gen_worker — clean, 231 files
  • ruff check src/gen_worker tests/test_cell_declare_verify_split_pgw988.py — clean
  • scripts/lint_http_timeouts.py, scripts/lint_unreached_surface.py,
    scripts/lint_config_reads.py — all exit 0

Release

No version bump in this PR. This needs to reach the fleet: unadoptable cells
cost a full compile per cold boot on every serving pod, so it should ride the
next release train (0.93.2, already queued for the attempt-24 gate hardcut and
ck5→ck6) rather than be published solo. Recorded in the tracker.

Closes pgw#988.

P0 regression on master, found by the pgw#978 rig and not by the lane that
caused it. th#1645 shrank the cell publish declare to control-plane size by
stripping the four envelope blocks that grow with the model — `entries` among
them. That was right about SIZE and it fixed a real 413 that had refused every
AOT cell publish thirty-two times. But `aot_cells._candidates` verifies the hub
LISTING metadata, and a listing row's metadata IS the declare body, with the
FULL `aot_serve.verify` — entries map and all. So from `c2e52f5f` every AOT cell
published by any worker was rejected on every pod as

    verify:malformed declared contract: metadata declares no entries map

and a pod that finds no cell mints its own. The fleet re-minted on every cold
boot, forever, with a typed reject class that named the cause exactly and a
symptom that presented as cost rather than as an error.

Restoring `entries` to the declare would re-create th#1645, so the fix is the
split the transport already implies. `aot_serve.verify_declared` rules on
`DECLARED_AXES` — format, kind, the code-only flag, sm/torch/cuda, host ISA,
family — everything a bounded control-plane declare carries, and therefore
everything discovery may refuse a cell for before it moves a byte.
`verify_contract` rules on the `entries` map, which is unbounded in the model,
rides INSIDE the artifact, and is verified by `stage_artifact` on the unpacked
`metadata.json` — where `aot_serve` has always read it from in order to serve.
`verify` is both, for callers holding a full envelope (`stage_artifact`).
Nothing is checked less than it was; one check moved to the side of the fetch
that actually has the data, and the pre-download filter still refuses every axis
a declare CAN carry, by its own class, with no bytes moved.

The defect class is the one pgw#985 names a layer up: two independent
computations of one fact — what a declare must carry — with no single authority,
so a change to one side could not see the other. There is an authority now.
`aot_cells.DECLARE_CONTRACT_KEYS` is every key the pre-download filter reads off
a declare, and `fleet_cells.assert_declare_contract` checks at import that
nothing `control_plane_metadata` strips appears in it. The next block that moves
fails every test run rather than a fleet.

`tests/test_cell_declare_verify_split_pgw988.py` is the rig's own seam as a CI
row, minus the card: a real `CellPublisher.publish` over a real socket into a
hub that stores the declare it actually received, then a real
`aot_cells.discover` reading that same body back off a real checkpoint listing
and pulling the real artifact bytes the publisher uploaded. Nothing between the
two halves is stubbed, because the defect lived exactly between them — each half
was correct alone. Every existing discovery test hands `_candidates` a full
envelope no publisher has ever put on the wire, which is why the whole suite
stayed green through the regression. RED on the unfixed tree: 7/7 fail with the
verbatim reject class above. GREEN here.

Gates: 3300 passed, 37 skipped, 1 xfailed (tests/); 21 passed (tests_v2/); mypy
clean over 231 files; ruff clean; lint_http_timeouts, lint_unreached_surface and
lint_config_reads all exit 0.
@PaulFidika

Copy link
Copy Markdown
Contributor Author

Coordination note: PR #512 (the 0.93.2 train) titles itself as including 'the declare/verify contract is split' — the same pgw#988 fix this PR carries. Whichever lands first: if #512 lands, verify content parity with this branch (tests/test_cell_declare_verify_split_pgw988.py must be present and green) then close this PR as superseded; if this lands first, #512 should rebase. Do not merge both blindly. Tracker record: pgw#988 section, commit f4b22cdd.

@PaulFidika PaulFidika closed this Aug 7, 2026
@PaulFidika PaulFidika reopened this Aug 7, 2026
@PaulFidika
PaulFidika merged commit fdbd4b2 into master Aug 7, 2026
2 checks passed
@PaulFidika
PaulFidika deleted the 988-declare-verify-split branch August 7, 2026 00:26
PaulFidika added a commit that referenced this pull request Aug 7, 2026
…sure leaves the key, and is_key stops pinning a scheme

REBUILT on fdbd4b2. PR #513 landed pgw#988 independently while this branch was
blocked on the signature rule, with materially the same design (verify_declared /
verify_contract, wired into the staged path at aot_serve.py:922). That
implementation is canonical; this branch drops its duplicate pgw#988 work rather
than contest a merged sibling's, and their tests own that ground. What remains is
pgw#990 only, and master still carries every line of it.

THE DEFECT. `aot_cells.prefer_aot()` (`Settings.compile_prefer_aot` /
`GEN_WORKER_PREFER_AOT`, default OFF) stood in front of TWO call sites:
`fleet_cells.enable_compiled`'s discovery branch, and `fleet_cells.mint_recipe`'s
first line, which returned RECIPE_DYNAMO whenever it was off. It reached pods
through a release-scoped endpoint_env entry that had to both match the release
(applies_to) AND be declared by it; neither held for any release after bdf54080.
So the forge minted kind=torch-inductor-cache cells that AOT discovery rejects by
name, and no serving pod ever looked for a cell. Last aot_cell_discovery event of
any kind on the dev fleet: 2026-08-02. Three A1 attempts ran un-armed.

The gate is DELETED, not re-pointed: Settings field, env name, loader mapping,
predicate, both call sites. provision's lifted-LoRA install now keys off the
artifact's own bucket (it was `bucket and compile_prefer_aot`, so an adopted
bucket-bearing cell reached aot_serve.enable UNLIFTED on exactly the pods the flag
had not reached). A pod still handed the retired env name reports it as
unrecognised owned env.

PROVEN ON HARDWARE 2026-08-06, pod zqwhoypxefadug (L40S, gen-worker 0.93.2):
  aot_cell_discovery/miss  family=sdxl lane=w8a8-lora64 ... not_an_aot_cell=5
  self_mint_started/aot    family=sdxl recipe=aot key=ck6-f512ec0e... lane=w8a8-lora64
First recipe=aot this program has recorded, and the first discovery event since
2026-08-02 — including discovery correctly refusing the previous dynamo cell.

code_closure LEAVES THE KEY; ck5 -> ck6. Identity is the COMPUTATION (traced graph
x sm x toolchain x env_seal); code hashes are a memo. A 147-file content hash
re-keyed every cell in the fleet for edits that cannot change a traced graph.
Still recorded, still driving the local re-trace memo, no longer identity.

is_key BECOMES SCHEME-AGNOSTIC, byte-identical to tensorhub's
compilecache.IsCellKey and for the reason th#1183 gives. This is load-bearing for
the ck6 bump: a ck5-pinned tree refuses every ck6 cell as `unreadable_cell_key`
before consulting any axis, which would strand the cells 0.93.2 workers mint.
Verified by running a 0.93.2-shaped declare through the current filter — which
also confirmed the extra bounded summary fields 0.93.2 declares are tolerated by
master's contract, so nothing strands on those either.

A GATE THAT CHANGES THE RECIPE MUST SAY SO. The deleted flag was the only one of
mint_recipe's paths that returned RECIPE_DYNAMO without calling _decline, so it
emitted nothing; the sole tell was the word "dynamo" inside a SUCCESS event's
detail. An AST guard now asserts the only `return RECIPE_DYNAMO` inside
mint_recipe is the one in _decline, which emits before returning. RED-verified
against v0.93.1.

Gates on this tree: mypy clean (231 files), ruff clean, http-timeout and
config-read guards pass, and 92 tests green across this lane's files PLUS the
sibling's test_cell_declare_verify_split_pgw988.py.

No version bump and no CHANGELOG edit: master bumps at a cut and lanes write
changelog.d fragments. These changes already shipped in the 0.93.2 tag, cut from
v0.93.1 so the A1 pod run had a pre-v1 wheel; this is how they reach master.
PaulFidika added a commit that referenced this pull request Aug 7, 2026
…sure leaves the key, and is_key stops pinning a scheme (#512)

REBUILT on fdbd4b2. PR #513 landed pgw#988 independently while this branch was
blocked on the signature rule, with materially the same design (verify_declared /
verify_contract, wired into the staged path at aot_serve.py:922). That
implementation is canonical; this branch drops its duplicate pgw#988 work rather
than contest a merged sibling's, and their tests own that ground. What remains is
pgw#990 only, and master still carries every line of it.

THE DEFECT. `aot_cells.prefer_aot()` (`Settings.compile_prefer_aot` /
`GEN_WORKER_PREFER_AOT`, default OFF) stood in front of TWO call sites:
`fleet_cells.enable_compiled`'s discovery branch, and `fleet_cells.mint_recipe`'s
first line, which returned RECIPE_DYNAMO whenever it was off. It reached pods
through a release-scoped endpoint_env entry that had to both match the release
(applies_to) AND be declared by it; neither held for any release after bdf54080.
So the forge minted kind=torch-inductor-cache cells that AOT discovery rejects by
name, and no serving pod ever looked for a cell. Last aot_cell_discovery event of
any kind on the dev fleet: 2026-08-02. Three A1 attempts ran un-armed.

The gate is DELETED, not re-pointed: Settings field, env name, loader mapping,
predicate, both call sites. provision's lifted-LoRA install now keys off the
artifact's own bucket (it was `bucket and compile_prefer_aot`, so an adopted
bucket-bearing cell reached aot_serve.enable UNLIFTED on exactly the pods the flag
had not reached). A pod still handed the retired env name reports it as
unrecognised owned env.

PROVEN ON HARDWARE 2026-08-06, pod zqwhoypxefadug (L40S, gen-worker 0.93.2):
  aot_cell_discovery/miss  family=sdxl lane=w8a8-lora64 ... not_an_aot_cell=5
  self_mint_started/aot    family=sdxl recipe=aot key=ck6-f512ec0e... lane=w8a8-lora64
First recipe=aot this program has recorded, and the first discovery event since
2026-08-02 — including discovery correctly refusing the previous dynamo cell.

code_closure LEAVES THE KEY; ck5 -> ck6. Identity is the COMPUTATION (traced graph
x sm x toolchain x env_seal); code hashes are a memo. A 147-file content hash
re-keyed every cell in the fleet for edits that cannot change a traced graph.
Still recorded, still driving the local re-trace memo, no longer identity.

is_key BECOMES SCHEME-AGNOSTIC, byte-identical to tensorhub's
compilecache.IsCellKey and for the reason th#1183 gives. This is load-bearing for
the ck6 bump: a ck5-pinned tree refuses every ck6 cell as `unreadable_cell_key`
before consulting any axis, which would strand the cells 0.93.2 workers mint.
Verified by running a 0.93.2-shaped declare through the current filter — which
also confirmed the extra bounded summary fields 0.93.2 declares are tolerated by
master's contract, so nothing strands on those either.

A GATE THAT CHANGES THE RECIPE MUST SAY SO. The deleted flag was the only one of
mint_recipe's paths that returned RECIPE_DYNAMO without calling _decline, so it
emitted nothing; the sole tell was the word "dynamo" inside a SUCCESS event's
detail. An AST guard now asserts the only `return RECIPE_DYNAMO` inside
mint_recipe is the one in _decline, which emits before returning. RED-verified
against v0.93.1.

Gates on this tree: mypy clean (231 files), ruff clean, http-timeout and
config-read guards pass, and 92 tests green across this lane's files PLUS the
sibling's test_cell_declare_verify_split_pgw988.py.

No version bump and no CHANGELOG edit: master bumps at a cut and lanes write
changelog.d fragments. These changes already shipped in the 0.93.2 tag, cut from
v0.93.1 so the A1 pod run had a pre-v1 wheel; this is how they reach master.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant