Skip to content

pgw#1009 (P1, catalog-blocking): the clone producer's checkpoint facts reach the publish declare - #527

Open
PaulFidika wants to merge 1 commit into
masterfrom
pgw1009-clone-classification
Open

pgw#1009 (P1, catalog-blocking): the clone producer's checkpoint facts reach the publish declare#527
PaulFidika wants to merge 1 commit into
masterfrom
pgw1009-clone-classification

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

Supersedes #526 (closed): that PR's branch carried id 1008, which the arm-gate
security lane claimed concurrently. Same commit, renumbered to pgw#1009.

The bug

th#1411 refuses a v2 publish into a repo whose live rows already carry
classification unless the request restates objective / distilled
(a v2 publish mints a new checkpoint identity and inherits nothing, th#1400).

run_clone accepts both facts (pgw#654) and spends them only on
apply_objective_scheduler_config — the scheduler stamp written inside the
produced tree. They never reach publish_v2, so the declare goes out bare.

publish_flavors (convert/publish.py) has passed both since pgw#654. This
call site is the one that did not, and git log -S "objective=objective_fact"
shows it never has.

Consequence

clone-huggingface cannot publish into any mirror the catalog already
serves.
Not "sometimes" — every classified repo, every time, at declare,
before a byte moves. Mirrors are exactly the repos that get classified, so the
producer that creates them cannot update them.

Live cost — ie#609: master's tensorhub/wan22-t2v-a14b is missing
transformer_2/ (the mirror was split into source_include legs and the third
leg was never submitted). The repair is one more leg:

{"huggingface_repo": "Wan-AI/Wan2.2-T2V-A14B-Diffusers",
 "source_revision": "5be7df96…",
 "destination_repo": "tensorhub/wan22-t2v-a14b",
 "source_include": ["model_index.json", "transformer_2/*"],
 "overwrite_repo": false, "destination_repo_tags": ["prod"],
 "objective": "flow", "distilled": false}

Refused twice on the master standing stack — 074a2c19-5988-409d-a09b-9362e90cacaa
(facts omitted) and 362957ad-ee57-4927-9611-ac9a04cc0bf0 (facts stated, same
refusal, which is what proved they never reach the wire) — both
400 classification_required … predecessor sha256:3b57e96996aa…. Every wan-2.2
function on the authoritative catalog is down until this ships.

The fix

Two kwargs on the publish_v2(...) call in clone.py, matching
publish_flavors. objective_fact / distilled_fact are the already
validated, already normalized locals.

Tests

tests/convert/test_clone_classification_pgw1009.py, hermetic against the
shared fake hub, driving the real run_clone:

  • test_declared_facts_reach_the_publish_declare — the ie#609 repair shape
    (merge leg into a classified mirror): the declare carries
    objective: "flow", distilled: false, mode: "merge".
  • test_undeclared_objective_stays_off_the_wire — an unstated objective is
    omitted rather than authored; distilled is a plain bool on the input
    struct, so an undeclared clone honestly states false.

Revert-turns-red: both fail with the two kwargs removed (verified).
Full tests/convert/: 74 passed, 2 skipped, 1 xfailed.

Deploy chain this unblocks

gen-worker release → training-endpoints/conversion pin (today
gen-worker==0.91.4; master's deployed conversion:prod 0.8.22 runs 0.91.4)
→ rebuild + redeploy conversion → re-run the ie#609 repair leg.

Tracker: pgw#1009, ie#609.

🤖 Generated with Claude Code

…s reach the publish declare

th#1411 refuses a v2 publish into a repo whose live rows carry classification
unless the request restates objective/distilled. run_clone spent the caller's
facts only on apply_objective_scheduler_config (the scheduler stamp inside the
produced tree) and dropped them before publish_v2 — so a clone-huggingface into
ANY already-classified mirror dies at declare, before a byte moves.

publish_flavors has passed both since pgw#654; this call site was the one that
did not.

Live cost (ie#609): master's tensorhub/wan22-t2v-a14b is missing transformer_2
because the mirror plan's third source_include leg was never run, and the
repair leg — one clone-huggingface merge, correct payload, objective='flow',
distilled=false — was refused twice at declare, 400 classification_required,
predecessor sha256:3b57e969. Every wan-2.2 function on the master catalog stays
down until this ships.

Revert-turns-red: both tests in tests/convert/test_clone_classification_pgw1009.py
fail with the two kwargs removed. Full tests/convert/: 74 passed, 2 skipped,
1 xfailed.

(Filed as pgw#1008 first; that id was taken concurrently by the arm-gate
security lane, so this is 1009.)
@PaulFidika

Copy link
Copy Markdown
Contributor Author

Supporting evidence that this is an omission from the th#1411 sweep, not a design choice:

every other publishing function in training-endpoints/conversion carries the same
# pgw#654/th#1411 checkpoint-fact override; unset = publish_flavors … note and threads the
facts through — fuse.py, transform.py, quant/bnb.py, quant/modelopt.py, quant/svdq.py,
lora_card.py, prompt_corpus.py, _common.py. All of them publish through
publish_flavors, which was fixed.

mirror.py is the only one that publishes through run_clone, and it is the only one still
broken. It is also the only producer that ingests from outside the catalog, so there is no
other conversion function a caller can fall back to: clone-huggingface / clone-civitai are
the sole path from an upstream repo into a tensorhub mirror.

@PaulFidika

Copy link
Copy Markdown
Contributor Author

Release vehicle — verified at source, so whoever merges this does not have to guess.

  • v0.93.4 is already tagged at 34124789 and awaiting its CI-on-tag publish run.
    This commit is not an ancestor of it (git merge-base --is-ancestor → false), and
    34124789:src/gen_worker/convert/clone.py still has the bare publish_v2(...) call.
    PyPI's newest is 0.93.3. So this fix lands in 0.93.5 or later.
  • The consumer that needs it is training-endpoints/conversion. te#167 PR B
    (167-repin-093-conversion) is already re-pointed at gen-worker 0.93.4 + conversion 0.8.29+.
    If PR B lands on 0.93.4 it will not carry this fix and a second conversion bump is owed.
  • Master's deployed conversion:prod 0.8.22 runs 0.91.4 (read off the pod's
    worker_boot_phases.gen_worker_version, not off the checkout), so the endpoint is behind
    either way.
  • te#167's own standing warning applies to the redeploy: do not deploy conversion mid-eval
    — check te#165's tracker tail for an in-flight slice first.

ie#609 (the master wan-2.2 outage) stays open until that chain completes. Its other three items
are done: the five wrong-tree bindings are rebound live on master, disk_gb is corrected
(inference-endpoints #267, merged), and the root cause is banked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant