Skip to content

pgw#999 (P0): the classified adopt refusal reaches the wire, and the rig covers the gate that produced it - #523

Merged
PaulFidika merged 2 commits into
masterfrom
999-adopt-reason
Aug 7, 2026
Merged

pgw#999 (P0): the classified adopt refusal reaches the wire, and the rig covers the gate that produced it#523
PaulFidika merged 2 commits into
masterfrom
999-adopt-reason

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

Attempt 26's sdxl mint succeeded — 36/36 entries exported, compiled, sealed, finalized — and was then refused by the runtime that had just built it. Three abort events fired and not one named a cause. 2 h 45 m and $2.72 of L40S bought the word "something", and the pod is gone.

The discard, exactly

provision.arm_aot returns a classified AdoptOutcome; aot_serve raises AdoptError with a reason token. fleet_cells.adopt_delegated_mint spent both on:

armed = bool(provision.arm_aot(...))

…and its except Exception branch wrote the reason to a logger no pod exposes. That is the worker-errors-to-orchestrator defect class verbatim.

The fix

phase is the countable column, so it carries the class — the convention self_mint_skipped already uses — and the detail quotes the gate's own sentence. Every branch classifies:

branch phase
returned miss its own reason (contract_invalid, no_arm_for_mode, numerics_refused, …)
raised AdoptError .reason (constants_constant_unresolved, …)
raised anything else the exception TYPE, not a flattened token
CellSelectionBugError cell_selection_bug (th#883's loud class, kept separate)
falsy arm, no classification unclassified_arm_refusal — never a blank phase, which reads as "no reason exists"

The classification is recorded on the pending, so DelegatedResult.reason and the executor's terminal error quote the one string produced where it was known, instead of three vocabularies for one fact.

Nine tests, all RED at HEAD, all GREEN here — verified by running them against the unpatched sources, not assumed.

The rig gap this found

task rig:micro's parity leg armed through aot_serve.enable. The delegated mint arms through provision.arm_aot, which additionally does the mode route, the lifted-LoRA install for a bucket-bearing cfg, and the numerics gate. The green cycle reported yesterday did not cover the path that refused sdxl. It does now, and stays green: 26.0 s, three entries, parity 7.15e-07.

Diagnosis so far — no pod, real path

Reproduced end to end through the real adopt_delegated_mint: a parent whose module does not expose an FQN the child traced is refused constants_constant_unresolved, naming the FQN (proj_in.weight (source=state_dict)).

Two hypotheses ruled out, not merely suspected:

  • extra per-layer buffers are harmless to constant binding (measured — a parent with +16 weight_scale buffers adopts cleanly);
  • the SDK's w8a8 Linear preserves the weight FQN (register_buffer("weight", …)) and only adds weight_scale/input_scale, so sdxl's quantized lane does not refuse by a renamed weight.

lane_unavailable is not reachable from this path at all: verify_declared gates on format, kind, package_constants_in_so, sm/torch/cuda, host_isa and family — and on no execution lane anywhere.

…overs the gate that produced it

Attempt 26's sdxl mint SUCCEEDED — 36/36 entries exported, compiled, sealed
and finalized — and was then refused by the runtime that had just built it.
Three abort events fired and not one named a cause:

  delegated_adopt_failed : "produced a cell this runtime could not adopt"
  delegated_no_cell      : "produced no adoptable cell (the child's cell did
                            not adopt on this runtime)"
  error                  : "delegated mint produced no advertisable cell"

The cause existed in-process. `provision.arm_aot` returns a classified
`AdoptOutcome` and `aot_serve` raises `AdoptError` with a reason token, and
`adopt_delegated_mint` spent both on `bool(...)`. 2 h 45 m and $2.72 of L40S
bought the word "something", and the pod is gone.

THE FIX. `phase` is the countable column, so it carries the CLASS — the
convention `self_mint_skipped` has always used — and the detail quotes the
gate's own sentence. Every branch classifies: a returned miss keeps its
reason, a raised `AdoptError` keeps `.reason`, an unclassified exception is
named by its TYPE rather than flattened into one useless token, and a falsy
arm that classified nothing says `unclassified_arm_refusal` instead of
emitting a blank phase that reads as "no reason exists". The classification is
recorded on the pending, so `DelegatedResult.reason` and the executor's
terminal error quote the ONE string produced where it was known, rather than
three vocabularies for one fact. The `except Exception` branch that wrote to a
logger no pod exposes now emits.

Nine tests, all RED at HEAD, all GREEN here — verified by running them against
the unpatched sources, not assumed.

THE RIG GAP THIS FOUND. `task rig:micro`'s parity leg armed through
`aot_serve.enable`; the delegated mint arms through `provision.arm_aot`, which
additionally does the mode route, the lifted-LoRA install for a bucket-bearing
cfg, and the numerics gate. So the green cycle I reported yesterday did not
cover the path that refused sdxl. It does now, and stays green: 26.0 s, three
entries, parity 7.15e-07.

DIAGNOSIS, so far, on the real path and with no pod. Reproduced end to end:
a parent whose module does not expose an FQN the child traced is refused
`constants_constant_unresolved`, naming the FQN. Two hypotheses are now RULED
OUT rather than suspected: extra per-layer buffers are harmless to constant
binding (measured), and the SDK's w8a8 Linear PRESERVES the `weight` FQN
(register_buffer("weight", ...)) and only adds `weight_scale`/`input_scale` —
so sdxl's quantized lane does not refuse by a renamed weight. `lane_unavailable`
is not reachable from this path at all: `verify_declared` gates on format,
kind, package_constants_in_so, sm/torch/cuda, host_isa and family, and on no
execution lane anywhere.
The same discard as the parent commit, one frame deeper, on exactly the
bucket-bearing path a `w8a8-lora64` family takes — which is the lane attempt
26 was minting.

`arm_aot`'s bucket branch caught a failed `install_lifted_lora_forward`,
PREDICTED its own downstream symptom in the log message ("a lifted artifact
will refuse at assert_lifted_contract"), and then threw the cause away. So the
refusal that follows names the gate that noticed, never the install that
failed, and the root exists only in a logger no pod exposes.

Both facts now ride the refusal, in the order a reader needs them: what
refused, and what made it refuse. RED-verified against the unpatched
`provision.py` — the failing run's captured log shows the root cause sitting
in the WARNING that was its only home.
@PaulFidika
PaulFidika merged commit c3ce51d into master Aug 7, 2026
2 checks passed
@PaulFidika
PaulFidika deleted the 999-adopt-reason branch August 7, 2026 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant