pgw#999 (P0): the classified adopt refusal reaches the wire, and the rig covers the gate that produced it - #523
Merged
Merged
Conversation
…overs the gate that produced it
Attempt 26's sdxl mint SUCCEEDED — 36/36 entries exported, compiled, sealed
and finalized — and was then refused by the runtime that had just built it.
Three abort events fired and not one named a cause:
delegated_adopt_failed : "produced a cell this runtime could not adopt"
delegated_no_cell : "produced no adoptable cell (the child's cell did
not adopt on this runtime)"
error : "delegated mint produced no advertisable cell"
The cause existed in-process. `provision.arm_aot` returns a classified
`AdoptOutcome` and `aot_serve` raises `AdoptError` with a reason token, and
`adopt_delegated_mint` spent both on `bool(...)`. 2 h 45 m and $2.72 of L40S
bought the word "something", and the pod is gone.
THE FIX. `phase` is the countable column, so it carries the CLASS — the
convention `self_mint_skipped` has always used — and the detail quotes the
gate's own sentence. Every branch classifies: a returned miss keeps its
reason, a raised `AdoptError` keeps `.reason`, an unclassified exception is
named by its TYPE rather than flattened into one useless token, and a falsy
arm that classified nothing says `unclassified_arm_refusal` instead of
emitting a blank phase that reads as "no reason exists". The classification is
recorded on the pending, so `DelegatedResult.reason` and the executor's
terminal error quote the ONE string produced where it was known, rather than
three vocabularies for one fact. The `except Exception` branch that wrote to a
logger no pod exposes now emits.
Nine tests, all RED at HEAD, all GREEN here — verified by running them against
the unpatched sources, not assumed.
THE RIG GAP THIS FOUND. `task rig:micro`'s parity leg armed through
`aot_serve.enable`; the delegated mint arms through `provision.arm_aot`, which
additionally does the mode route, the lifted-LoRA install for a bucket-bearing
cfg, and the numerics gate. So the green cycle I reported yesterday did not
cover the path that refused sdxl. It does now, and stays green: 26.0 s, three
entries, parity 7.15e-07.
DIAGNOSIS, so far, on the real path and with no pod. Reproduced end to end:
a parent whose module does not expose an FQN the child traced is refused
`constants_constant_unresolved`, naming the FQN. Two hypotheses are now RULED
OUT rather than suspected: extra per-layer buffers are harmless to constant
binding (measured), and the SDK's w8a8 Linear PRESERVES the `weight` FQN
(register_buffer("weight", ...)) and only adds `weight_scale`/`input_scale` —
so sdxl's quantized lane does not refuse by a renamed weight. `lane_unavailable`
is not reachable from this path at all: `verify_declared` gates on format,
kind, package_constants_in_so, sm/torch/cuda, host_isa and family, and on no
execution lane anywhere.
The same discard as the parent commit, one frame deeper, on exactly the
bucket-bearing path a `w8a8-lora64` family takes — which is the lane attempt
26 was minting.
`arm_aot`'s bucket branch caught a failed `install_lifted_lora_forward`,
PREDICTED its own downstream symptom in the log message ("a lifted artifact
will refuse at assert_lifted_contract"), and then threw the cause away. So the
refusal that follows names the gate that noticed, never the install that
failed, and the root exists only in a logger no pod exposes.
Both facts now ride the refusal, in the order a reader needs them: what
refused, and what made it refuse. RED-verified against the unpatched
`provision.py` — the failing run's captured log shows the root cause sitting
in the WARNING that was its only home.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attempt 26's sdxl mint succeeded — 36/36 entries exported, compiled, sealed, finalized — and was then refused by the runtime that had just built it. Three abort events fired and not one named a cause. 2 h 45 m and $2.72 of L40S bought the word "something", and the pod is gone.
The discard, exactly
provision.arm_aotreturns a classifiedAdoptOutcome;aot_serveraisesAdoptErrorwith a reason token.fleet_cells.adopt_delegated_mintspent both on:…and its
except Exceptionbranch wrote the reason to a logger no pod exposes. That is theworker-errors-to-orchestratordefect class verbatim.The fix
phaseis the countable column, so it carries the class — the conventionself_mint_skippedalready uses — and the detail quotes the gate's own sentence. Every branch classifies:contract_invalid,no_arm_for_mode,numerics_refused, …)AdoptError.reason(constants_constant_unresolved, …)CellSelectionBugErrorcell_selection_bug(th#883's loud class, kept separate)unclassified_arm_refusal— never a blank phase, which reads as "no reason exists"The classification is recorded on the pending, so
DelegatedResult.reasonand the executor's terminal error quote the one string produced where it was known, instead of three vocabularies for one fact.Nine tests, all RED at HEAD, all GREEN here — verified by running them against the unpatched sources, not assumed.
The rig gap this found
task rig:micro's parity leg armed throughaot_serve.enable. The delegated mint arms throughprovision.arm_aot, which additionally does the mode route, the lifted-LoRA install for a bucket-bearing cfg, and the numerics gate. The green cycle reported yesterday did not cover the path that refused sdxl. It does now, and stays green: 26.0 s, three entries, parity 7.15e-07.Diagnosis so far — no pod, real path
Reproduced end to end through the real
adopt_delegated_mint: a parent whose module does not expose an FQN the child traced is refusedconstants_constant_unresolved, naming the FQN (proj_in.weight (source=state_dict)).Two hypotheses ruled out, not merely suspected:
+16 weight_scalebuffers adopts cleanly);weightFQN (register_buffer("weight", …)) and only addsweight_scale/input_scale, so sdxl's quantized lane does not refuse by a renamed weight.lane_unavailableis not reachable from this path at all:verify_declaredgates on format, kind,package_constants_in_so, sm/torch/cuda, host_isa and family — and on no execution lane anywhere.