pgw#1001 (P0): a bucket-bearing cell can be adopted by the runtime that minted it — three defects, ruled by invariant - #531
Merged
Conversation
…overs the gate that produced it
Attempt 26's sdxl mint SUCCEEDED — 36/36 entries exported, compiled, sealed
and finalized — and was then refused by the runtime that had just built it.
Three abort events fired and not one named a cause:
delegated_adopt_failed : "produced a cell this runtime could not adopt"
delegated_no_cell : "produced no adoptable cell (the child's cell did
not adopt on this runtime)"
error : "delegated mint produced no advertisable cell"
The cause existed in-process. `provision.arm_aot` returns a classified
`AdoptOutcome` and `aot_serve` raises `AdoptError` with a reason token, and
`adopt_delegated_mint` spent both on `bool(...)`. 2 h 45 m and $2.72 of L40S
bought the word "something", and the pod is gone.
THE FIX. `phase` is the countable column, so it carries the CLASS — the
convention `self_mint_skipped` has always used — and the detail quotes the
gate's own sentence. Every branch classifies: a returned miss keeps its
reason, a raised `AdoptError` keeps `.reason`, an unclassified exception is
named by its TYPE rather than flattened into one useless token, and a falsy
arm that classified nothing says `unclassified_arm_refusal` instead of
emitting a blank phase that reads as "no reason exists". The classification is
recorded on the pending, so `DelegatedResult.reason` and the executor's
terminal error quote the ONE string produced where it was known, rather than
three vocabularies for one fact. The `except Exception` branch that wrote to a
logger no pod exposes now emits.
Nine tests, all RED at HEAD, all GREEN here — verified by running them against
the unpatched sources, not assumed.
THE RIG GAP THIS FOUND. `task rig:micro`'s parity leg armed through
`aot_serve.enable`; the delegated mint arms through `provision.arm_aot`, which
additionally does the mode route, the lifted-LoRA install for a bucket-bearing
cfg, and the numerics gate. So the green cycle I reported yesterday did not
cover the path that refused sdxl. It does now, and stays green: 26.0 s, three
entries, parity 7.15e-07.
DIAGNOSIS, so far, on the real path and with no pod. Reproduced end to end:
a parent whose module does not expose an FQN the child traced is refused
`constants_constant_unresolved`, naming the FQN. Two hypotheses are now RULED
OUT rather than suspected: extra per-layer buffers are harmless to constant
binding (measured), and the SDK's w8a8 Linear PRESERVES the `weight` FQN
(register_buffer("weight", ...)) and only adds `weight_scale`/`input_scale` —
so sdxl's quantized lane does not refuse by a renamed weight. `lane_unavailable`
is not reachable from this path at all: `verify_declared` gates on format,
kind, package_constants_in_so, sm/torch/cuda, host_isa and family, and on no
execution lane anywhere.
The same discard as the parent commit, one frame deeper, on exactly the
bucket-bearing path a `w8a8-lora64` family takes — which is the lane attempt
26 was minting.
`arm_aot`'s bucket branch caught a failed `install_lifted_lora_forward`,
PREDICTED its own downstream symptom in the log message ("a lifted artifact
will refuse at assert_lifted_contract"), and then threw the cause away. So the
refusal that follows names the gate that noticed, never the install that
failed, and the root exists only in a logger no pod exposes.
Both facts now ride the refusal, in the order a reader needs them: what
refused, and what made it refuse. RED-verified against the unpatched
`provision.py` — the failing run's captured log shows the root cause sitting
in the WARNING that was its only home.
…l — and the plain variant's parity REGRESSED under the rename. Do not merge.
CHECKPOINT, NOT A CANDIDATE. Pushed so nothing is lost; the plain variant is
RED and the cause is not yet root-caused.
WHAT THIS BUYS (all measured locally, ~15-25 s/cycle, no pod):
* A `micro-lora` vehicle at `lora_bucket=64` — sdxl's LORA half. The w8a8 half
is unreachable on this box (pgw#983: no CUDA, so `w8a8_gemm_mode()` is "" and
the module class refuses), but the bucket axis is lane-agnostic (gw#558
covers plain resident Linears) and it is the axis `arm_aot` grows an entire
extra gate pair for.
* It REPRODUCES a real, classified arm refusal in ~20 s, and walking it forward
found TWO genuine `arm_aot` defects, each of which alone makes a
bucket-bearing cell unadoptable by the runtime that minted it:
1. a packed multi-entry cell records its targets PER ENTRY and carries NO
top-level `targets`/`module` (measured: both None on a real 5-entry
lora64 cell), so `module_name` came out "" and the lifted-binding
install was SILENTLY SKIPPED -> `lifted_inputs_unbindable`;
2. with that fixed, the naive `targets[0]` picks `decoder` (dict order),
and installing a lifted forward on a module with no branch container
fails by name -> still `lifted_inputs_unbindable`, now with the root
visible ONLY because of the parent commit's change. `branch_targets`
is the authority on which module is the denoiser.
With both fixed the arm advances to `numerics_refused` — the next gate.
WHY IT IS RED, STATED HONESTLY:
Renaming the family's denoiser attribute `denoiser` -> `transformer` (required:
`branch_targets` only recognises transformer/unet/transformer_2/dit, so the
LoRA lane could not apply to a module named `denoiser`) turned the numerics
gate ON for this family for the first time. The gate now reports HEALTHY while
the rig's own parity reports 0.151 on the transformer arms and 4.9e-07 on the
decoder. Two measurements of the same claim disagree, and I have NOT
root-caused it. Taking the eager reference from a pristine second instance did
not change it, so it is not reference contamination.
That disagreement is itself a finding worth keeping — but it is not a merge.
…s lying, the gate is sound; and the lora64 red is now fully diagnosed VERDICT ON THE DISAGREEMENT (gate=HEALTHY vs rig-parity=0.151). The RIG was wrong. Renaming this family's denoiser `denoiser` -> `transformer` left the CACHED checkpoint tree writing `denoiser.*` keys while the new loader looked for `transformer.*` — under `strict=False`. Every lookup missed, the module loaded ZERO tensors and stayed randomly initialized, and two processes therefore held two DIFFERENT random transformers. The rig compared them and read 0.151. The decoder, whose prefix did not change, read 4.9e-07 — which is exactly the split that was reported, and it was the clue. `materialize()` is idempotent by CONFIG, and the config never mentioned the state-dict prefixes, so the rename could not invalidate the cache. THE GATE IS A WORKING INSTRUMENT — proven, not assumed. Could-it-have-failed test against one armed cell: untouched -> healthy (cosine 1.0); eager weights moved 1% -> healthy (0.99995); eager weights moved 25% -> `destroyed` (0.9728) and `degraded` (0.9862). It goes red on real divergence. Worth recording: the ladder is COSINE-based, so a uniform ~1% error passes — a real property of the instrument, not a defect, but not a max-abs bound either. FIXES, both of the silent-failure class this program keeps paying for: * the loader is STRICT and refuses by name when a prefix selects nothing, naming the prefixes the tree actually has. `strict=True` alone would not have caught it — an empty dict is just "missing keys" and says nothing about the layout. * the tree carries a LAYOUT_VERSION, so a component rename invalidates the cache instead of leaving bytes no loader can read. With true weights the plain variant is GREEN again at 7.15e-07, 13.4 s — the rig and the gate now agree, which is the state a parity claim requires. AND THE lora64 RED IS NOT A NUMERICS DIVERGENCE AT ALL. Per-axis probe: decoder healthy cosine=1.0 transformer/adapter=true,cfg=false healthy cosine=1.0 transformer/adapter=true,cfg=true healthy cosine=1.0 transformer/adapter=false,cfg=false eager_forward_failed transformer/adapter=false,cfg=true eager_forward_failed "the EAGER reference forward raised (ValidationError: the lifted LoRA argument 'lora_a' is missing)" `arm_aot` installs the lifted forward because the adapter=true entries need it; the installed forward then DEMANDS lora_a/lora_b on every call; the gate probes every packaged axis including the BRANCHLESS ones, whose feed correctly omits the adapter inputs (pgw#790: the branchless class declares none and refuses them); the eager reference — now the lifted forward — raises; the axis is unmeasurable; the whole cell is refused `numerics_refused`. So a bucket-bearing cell that packages BOTH adapter arms can never pass the numerics gate on a runtime where the lifted forward is installed. That is a structural contradiction between pgw#790's adapter fork and pgw#868's gate, and it is a strong candidate for what refused attempt 26's w8a8-lora64 cell. Filed as pgw#1001 defect 3; NOT patched here, because the fix is a design call (disarm the lift for a branchless probe, or keep a branchless path on the lifted forward) and this branch is already a checkpoint.
…nted it — three defects, ruled by invariant sdxl minted `w8a8-lora64` and the cell was refused by the pod that built it. This is that lane's LoRA half, reproduced locally in ~20 s a cycle on the pgw#997 micro family once pgw#999 made the refusal say its own name. Three defects, each of which ALONE made adoption impossible. 1. `arm_aot` resolved its lifted target from `meta["targets"]`/`meta["module"]`. A packed multi-entry cell carries NEITHER — measured on a real 5-entry lora64 cell, both are None and the targets live PER ENTRY. `module_name` came out "", the lifted-binding install was SILENTLY SKIPPED, and `aot_serve.enable` then refused the artifact it had just been handed with `lifted_inputs_unbindable`. 2. With that fixed, the naive `targets[0]` picks `decoder` (entry-name order), and installing a lifted forward on a module with no branch container fails by name. `lora_lifted.branch_targets` is the one authority on which module is the denoiser, and it is what chooses now. 3. THE INVARIANT — and this one is a SERVING bug, not a probe bug. `install_lifted_lora_forward` replaces `model.forward` WHOLESALE, so a plain call that worked the instant before the install RAISED the instant after it. Verified on the rig before choosing a fix: the identical feed returns (1, 1024, 4) before, and `ValidationError: the lifted LoRA argument 'lora_a' is missing` after. A branchless request falling back to eager on an ARMED pod hits that in production; it surfaced as `numerics_refused` only because the numerics gate is the thing that probes the branchless axis (pgw#790's adapter fork). Fixing the probe would have masked it. Ruled by invariant: ARMING A BUCKET MUST NOT ALTER THE SEMANTICS OF CALLS THAT DO NOT USE THE BUCKET. The lifted forward now takes the plain branch whenever BOTH operands are absent — after the positional-feed extraction, so it sees the pair however it arrived, and including keyword-style plain calls. Plain-call delta after install is EXACTLY 0.0. A half-supplied pair is still a caller error and still refuses. WHAT THIS DELIBERATELY RETIRES, measured rather than assumed. The old refusal doubled as a guard against CAPTURING an adapter class without its operands. Two facts retire it: tracing the BRANCHLESS arm with no operands is CORRECT (that arm is what an `adapter=false` entry IS), and a `torch.compiler.is_compiling()` guard to separate the cases reads FALSE under STRICT export — the mint's own mode — with both a deferred and an install-time-bound torch. A guard measured not to fire is worse than none, so it is not shipped. The protection that matters belongs at the mint, on the adapter arm's feed; filed as the follow-up rather than left implied. RESULT: `micro-lora` mints 5 entries, publishes, adopts, arms, passes the numerics gate and reaches parity 7.15e-07. The plain vehicle is unchanged at 7.15e-07. Also carried (pgw#999 follow-up, and the reason the numerics gate was wrongly suspected): the micro family's loader is STRICT and refuses by name when a prefix selects nothing, and the generated tree carries a LAYOUT_VERSION so a component rename invalidates the cache. Under `strict=False` a renamed prefix loaded ZERO tensors silently, two processes held two different random modules, and the rig read it as a 0.151 divergence. The gate was right all along — proven by a could-it-have-failed run: untouched healthy, +1% healthy (0.99995 cosine), +25% destroyed/degraded. It is a DIRECTIONAL-DEGRADATION instrument, not a max-abs bound. 5 tests, 4 RED at HEAD; the fifth guards the half-pair refusal this must not break. One pgw#725 assertion is deliberately CHANGED — its docstring records that it encoded the serving break.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sdxl minted
w8a8-lora64and the cell was refused by the pod that built it. This is that lane's LoRA half, reproduced locally in ~20 s a cycle on the pgw#997 micro family once pgw#999 made the refusal say its own name. Three defects, each of which ALONE made adoption impossible. Landing as one coherent change.1. The lifted target was looked for where the metadata does not put it
Measured on a real 5-entry lora64 cell:
meta["targets"] is Noneandmeta["module"] is None. The targets are recorded per entry. Somodule_namewas"", the lifted-binding install was silently skipped, andaot_serve.enablethen refused the artifact it had just been handed withlifted_inputs_unbindable. A bucket-bearing cell could never adopt on the runtime that minted it.2.
targets[0]is the wrong targetWith 1 fixed, the entry-name-ordered candidate list puts
decoderfirst, and installing a lifted forward on a module with no branch container fails by name.lora_lifted.branch_targetsis the one authority on which module is the denoiser, and it is what chooses now.3. THE INVARIANT — and this one is a SERVING bug, not a probe bug
install_lifted_lora_forwardreplacesmodel.forwardwholesale. Verified on the rig before choosing a fix:A branchless request falling back to eager on an armed pod hits that in production. It surfaced as
numerics_refusedonly because the numerics gate is the thing that probes the branchless axis (pgw#790's adapter fork) — so fixing the probe would have masked it.Ruled by invariant: arming a bucket must not alter the semantics of calls that do not use the bucket. The lifted forward now takes the plain branch whenever both operands are absent — after the positional-feed extraction, so it sees the pair however it arrived, keyword-style plain calls included. Plain-call delta after install is exactly 0.0. A half-supplied pair is still a caller error and still refuses.
What this deliberately retires, measured rather than assumed
The old refusal doubled as a guard against capturing an adapter class without its operands. Two facts retire it:
adapter=falseentry is — so the refusal was wrong for half the classes a bucket-bearing cell packages;torch.compiler.is_compiling()guard to separate the two reads False under STRICT export, the mint's own mode (measured with both a deferred and an install-time-bound torch; this module defersimport torchon purpose because build-time discovery has none). A guard measured not to fire is worse than none, so it is not shipped.The protection that matters — an adapter arm's feed must contain both operands — belongs at the mint, off the serving path. Filed as pgw#1001's follow-up rather than left implied.
Result
micro-loramints 5 entries, publishes, adopts, arms, passes the numerics gate, parity 7.15e-07. Plain vehicle unchanged at 7.15e-07.Also carried — and it is why the numerics gate was wrongly suspected
The micro family's loader is now strict and refuses by name when a prefix selects nothing, and the generated tree carries a
LAYOUT_VERSIONso a component rename invalidates the cache. Understrict=Falsea renamed prefix loaded zero tensors silently, two processes held two different random modules, and the rig read it as a 0.151 divergence.The gate was right all along, proven by a could-it-have-failed run: untouched → healthy (cosine 1.0); eager weights +1% → still healthy (0.99995); +25% →
destroyed(0.9728) /degraded(0.9862). It is a directional-degradation instrument, not a max-abs bound — worth saying wherever A1's parity verdict is quoted.Tests
5 new, 4 RED at HEAD (the fifth guards the half-pair refusal this must not break). One pgw#725 assertion is deliberately changed; its docstring records that it encoded the serving break.