Skip to content

pgw#1010: the JIT/dynamo recipe stops producing artifacts — it keeps serving, and loses its cell - #532

Open
PaulFidika wants to merge 4 commits into
masterfrom
1010-dynamo-no-publish
Open

pgw#1010: the JIT/dynamo recipe stops producing artifacts — it keeps serving, and loses its cell#532
PaulFidika wants to merge 4 commits into
masterfrom
1010-dynamo-no-publish

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

pgw#1010 — the JIT/dynamo recipe keeps serving and loses its cell

Paul's ratified reuse ruling — reuse = AOT cells only; JIT = intake mode, honest cold
boots
— composed with two facts already in the tree: aot_cells._candidates rejects
kind="torch-inductor-cache" by name, and nothing else has ever adopted one. A dynamo
cell therefore had zero possible consumers, and every one minted spent pod time, a
publish quota slot and platform storage on an artifact no pod could adopt (attempt 23's
ck5-a53e02a7… among them).

The cut. A family with no export declaration serves JIT INTAKE:
compile_cache.arm_jit_intake arms the declared targets cold-allowed and guarded, this
pod's own warmup compiles them, the pod serves compiled for its own life — and nothing is
captured, keyed, packed, published or owed. Deleted, not gated: there is no flag that
re-enables dynamo publishing.

Gone with it, each because it only ever built or shipped that cell:

  • compile_cache.begin_fleet_mint (→ arm_jit_intake, minus the capture dir) and
    finish_fleet_mint; _capture_forensics;
  • fleet_cells.finalize_self_mint, republish_after_shape_warm, and
    PendingSelfMint.recipe/.capture_dir (every pending is a delegated AOT mint now);
  • the executor's in-process/background mint driver, its boot-proof pack + gw#612 publish
    gate, and the shape-warm republisher;
  • mint_child's dynamo branch and _drain_router;
    mint_delegate._emit_jit_compile/_emit_warm_ledger;
  • GEN_WORKER_MINT_IN_PROCESS, with the shape it selected — pgw#995 recorded it as the
    last deletable behaviour switch and named its blocker (ten test sites forcing the shape
    through the executor); the shape's removal discharges that debt. With the process-global
    inductor cache-dir move gone, so are gw#608's delivered_cell_seeded gate, pgw#777's
    multi-execution-group refusal and the one-capture-per-process conflict.

Two consequences, decided rather than discovered later

  1. A mandatory (w8a8/w4a4) lane fails closed when the family declares no export
    (mandatory_lane_needs_a_cell). That lane serves only from a cell — the th#910 dispatch
    fence pins every request to an active compile incarnation — so an intake arm there would
    compile a whole boot for a pod that then refuses every request
    required_compile_missing (observed exactly that before the gate went in). This is the
    pre-self-mint posture restored and the one behaviour change to ratify: to serve a
    mandatory lane, a family must declare an export, because an AOT cell is the only cell.
  2. Intake stays visible. An intake-armed pipeline reports serving_mode=jit_cell with
    an empty served_cell_ref (compile_cache.is_compile_armed), binds the pgw#680
    guard-miss callbacks (previously gated on active_ref, which intake has none of — every
    guard miss on the platform would have gone off the wire), and emits th#1322's
    jit_compile duration from the boot warm window (the old emitter was the mint parent's).

Obligation settlement — verified in hub source, no hub change required

Obligations are seeded hub-side per (release_id, sku), never from a worker event, and
discharge is hard-gated on a real cell_store row (publishedCellSKUsByRelease /
publishedCellLanded, th#1644(c)/th#1645) — so a mint that publishes nothing cannot
discharge one. The typed no-artifact settlement already exists: mint_recipe's decline
emits self_mint_skipped with no_export_declaration / declaration_refused /
declaration_module_mismatch, all three in th#1628's DeterministicMintRefusalPhases, so
the obligation goes terminally refused instead of burning two 45-minute boot windows.
What the cut REMOVES hub-side is a false positive: discharge is kind-blind (cell_store
has no kind column), so a published dynamo cell used to settle an obligation that wanted
an AOT one. th#1661 is filed for the residue this exposes and does not fix (pull-by-key
demand for keys nothing can publish).

Proof

  • RED: on master, test_fleet_cells.py::test_finalize_packs_the_proven_capture_and_publishes_it
    passes — it packs a capture and publishes it with axes["kind"] == "torch-inductor-cache",
    through the production finalize_self_mint + CellPublisher. On this branch those
    functions do not exist. (The dynamo mint itself cannot be red-run locally: its cold arm
    requires CUDA, and no pods were used.)
  • GREEN, full machinery, local CPU: micro_mint_rig --vehicle micro --stage all
    handoff → child spawn → load → warm proof → torch.export + AOTInductor (3 entries) → seal
    → publish over 4 real HTTP calls to a local hub → a second process adopts the cell and
    parity-checks it: max|delta| = 7.15e-07 over 3 arms, 32.2 s. The surviving lane is
    intact end to end, publish included.
  • New guard: tests/test_dynamo_no_publish_pgw1010.py — an AST source guard (the intake
    branch cannot reach the seal/publish vocabulary; the vocabulary is gone where it was
    dynamo-only; intake computes no cell key) plus a behavioural guard driving the real
    arming policy (serves, owes nothing, publishes nothing, says so once with the phase the
    hub counts).
  • Suite re-aimed at the surviving shape rather than re-pointed: rigs that modelled the
    in-process capture now model the delegated AOT mint or JIT intake, and the rows whose
    subject WAS the capture are deleted with it.

…serving, and loses its cell

Paul's ratified reuse ruling (reuse = AOT cells only; JIT = intake mode, honest
cold boots) composed with two facts already in the tree: aot_cells rejects
kind="torch-inductor-cache" BY NAME, and nothing else has ever adopted one. A
dynamo cell had zero possible consumers, so every one minted spent pod time, a
publish quota slot and platform storage on an artifact no pod could adopt.

A family with no export declaration now serves JIT INTAKE: arm the declared
targets cold-allowed and guarded (compile_cache.arm_jit_intake), this pod's own
warmup compiles them, and nothing is captured, keyed, packed, published or owed.
Deleted, not gated.

Gone with it, each because it only ever built or shipped that cell:
begin_fleet_mint/finish_fleet_mint, fleet_cells.finalize_self_mint and
republish_after_shape_warm, the executor's in-process mint driver + boot-proof
pack + gw#612 publish gate + shape-warm republisher, mint_child's dynamo branch
and _drain_router, mint_delegate._emit_jit_compile/_emit_warm_ledger, and
compile_cache._capture_forensics.

GEN_WORKER_MINT_IN_PROCESS is deleted with the shape it selected — pgw#995 named
it the last deletable behaviour switch and named its blocker (ten test sites);
the shape's removal discharges that debt. With the process-global inductor
cache-dir move gone, so are gw#608's delivered-cell-seeded gate, pgw#777's
multi-group refusal and the one-capture-per-process conflict.

Two consequences made explicit rather than discovered later:
- a MANDATORY (w8a8/w4a4) lane serves only from a cell, so a family with no
  export declaration fails closed there (mandatory_lane_needs_a_cell) instead of
  compiling an intake arm every request would refuse required_compile_missing;
- an intake-armed pipeline reports serving_mode=jit_cell with an empty
  served_cell_ref, binds the pgw#680 guard-miss callbacks, and emits the th#1322
  jit_compile duration from the executor's proof window — otherwise the only JIT
  lane left would compile silently and off the wire.

Hub-side settlement verified and unchanged: obligations are seeded per
(release, sku), discharge already requires a real cell_store row
(th#1644/th#1645), and the typed no-artifact settlement is th#1628's
deterministic self_mint_skipped refusal (no_export_declaration et al). What the
cut removes hub-side is a false positive — discharge is kind-blind, so a
published dynamo cell used to settle an obligation that wanted an AOT one.
Filed th#1661 for the residue (pull-by-key demand nothing can satisfy).
Every rig that modelled the IN-PROCESS capture is either re-aimed at the
delegated AOT mint (which is now the only mint) or at JIT INTAKE (which is now
the only dynamo), and the rows whose subject was the capture itself — pack
refusals, partial-capture finalize, the two-lane union publish, the in-process
mint driver's routing branch — are deleted with it rather than re-pointed.

New: tests/test_dynamo_no_publish_pgw1010.py — the source guard (the intake
branch cannot reach the seal/publish vocabulary, the vocabulary is gone where
it was dynamo-only) plus the behavioural guard (a real arming-policy miss with
no export declaration serves, owes nothing, publishes nothing, and says so
once with the phase the hub counts).
The jit_compile duration event is emitted from the boot warm window and sited
OUTSIDE the proof block: an intake arm names no artifact, so proves_inductor is
false for it and anything inside that block is unreachable from the only JIT
lane left. compile_wall_seconds is now read unconditionally (a counter read),
because gating the measurement on an active artifact is what would have made
the intake compile the one nobody timed.
@PaulFidika
PaulFidika force-pushed the 1010-dynamo-no-publish branch from 2756c55 to 1cfa11b Compare August 7, 2026 09:16
The guard's permanent fallback now records the degrade on the SHARED failure
signal, and is_compile_armed reads it. Without this an intake pod that lost its
compiled lane would keep reporting serving_mode=jit_cell while every request ran
eager — the gw#586 class, one lane over.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant