Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions changelog.d/pgw958.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
- **pgw#958 (DESIGN-RULINGS §1.27(g)): the three cell-identity counters restart
at 1.** `cell_key.KEY_SCHEME` `ck6` -> `ck1`, `env_seal.SEAL_VERSION` `6` ->
`1`, `guard_closure.MANIFEST_VERSION` `3` -> `1`. Paul ruled on the collision
with pgw#990's `ck5` -> `ck6` bump on 2026-08-07 — *"the ck6 is wrong and can
be removed / deleted just fine. The v1 is the correct one"* — so `ck6` is
reverted rather than carried, and the cells it keyed join the purge as the
deletable artifacts they are. pgw#990's two INDEPENDENT decisions are
untouched and deliberately kept: `code_closure` stays out of the key, and
`is_key` stays scheme-AGNOSTIC (byte-identical to tensorhub's
`compilecache.IsCellKey`, for the reason th#1183 gives) — a foreign-scheme
token is still key-SHAPED and is ruled on by axes, never by its label. All
three counters feed the cell-key
digest, so this re-keys every cell in one cut rather than three; the failure
mode is a MISS (eager serve + demand + forge), never a refusal, and the hub's
`IsCellKey` validates key SHAPE only (th#1183), so `ck1-…` publishes and
resolves unchanged. Unlike the protocol enum, the lower numbers here were
really used and really minted cells, so re-issuing 1 is only honest once the
old corpus is gone: the reset is paired with a counted purge of every
pre-existing `ck1..ck6` cell row and its CAS objects — tensorhub migration
`20260806T064516Z-e68cfe98` (th#1636), applied to both standing stacks on
2026-08-06, with `cas-gc` the operator follow-up.
`COZY_CELL_EPOCH` remains the designed disown-everything lever; these
counters never move for recall reasons again. No ordering comparison anywhere
read any of the three — the only version compare in the tree is
`aot_resume.BANK_V`, already an equality check at 1. The
`aot_wrapper_split` telemetry label `version="v2"` became `lever="runimpl"`
(`runimpl_applied` / `runimpl_declined`): it names which split fired, and was
never a version.
2 changes: 1 addition & 1 deletion scripts/unreached_surface_baseline.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
# (`c47dc46`) gave all three their first production callers, the guard went
# red on STALE, and the lines came out. A symbol ceasing to be dead code is
# this guard's nicest failure mode, and the ratchet turned the right way.
# compile_cache.assert_closure_complete — the ck5 code-closure completeness
# compile_cache.assert_closure_complete — the code-closure completeness
# gate ("a dynamic import is hiding trace-relevant code from the recipe
# key"). Its only caller is a test.
# aot_serve.is_aot_artifact — its own docstring says "kind sniff for the
Expand Down
8 changes: 4 additions & 4 deletions src/gen_worker/aot_cells.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

Filter (SDXL-AOT-PILOT-RUNBOOK.md §3 F1):

* ``kind == aot-inductor`` with a stamped ck5 ``cell_key``;
* ``kind == aot-inductor`` with a stamped ``cell_key``;
* runtime-key check via ``aot_serve.verify_declared`` (sm, torch, cuda,
family, host ISA, code-only flag) — the DECLARE half only. The ``entries``
contract is unbounded in the model, does not ride the control-plane declare
Expand Down Expand Up @@ -88,7 +88,7 @@
#: that nothing it strips from the declare appears here — the check pgw#988
#: cost a day of unadoptable cells for not existing.
DECLARE_CONTRACT_KEYS = frozenset(aot_serve.DECLARED_AXES) | frozenset({
"cell_key", # ck5 identity + the artifact cache name
"cell_key", # cell identity + the artifact cache name
"weight_lane", # \_ the canonical execution lane this pipeline needs
"lora_bucket", # /
"sku", # selection PREFERENCE, never a filter (pgw#765)
Expand All @@ -108,7 +108,7 @@ class AdoptedAotCell:

family: str
cell_key: str
ref: str # "root/family-<f>#<stamped ck5 key>"
ref: str # "root/family-<f>#<stamped cell key>"
snapshot_digest: str # "sha256:<hex>" of the artifact tarball
artifact: Path

Expand Down Expand Up @@ -465,7 +465,7 @@ def _discover_inner(
if artifact is None:
return None
# From this moment the executor's kind dispatch (#734/#735) must
# recognize the ck5-flavored ref as an exported cell.
# recognize the key-flavored ref as an exported cell.
aot_serve.note_aot_key(key)
adopted = AdoptedAotCell(
family=family,
Expand Down
2 changes: 1 addition & 1 deletion src/gen_worker/aot_flatten.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ def name(self) -> str:
A mapping leaf takes its BARE KEY because that is the keyword the
pipeline's own forward uses; a sequence leaf takes ``<param>.<index>``.
This is the string the published contracts are keyed by and the one
``contract_digest`` folds into ck6, so it is fixed — pgw#994 adds the
``contract_digest`` folds into the key, so it is fixed — pgw#994 adds the
identity next to it rather than renaming 144 live checkpoints.
"""
name = self.param
Expand Down
4 changes: 2 additions & 2 deletions src/gen_worker/aot_mint.py
Original file line number Diff line number Diff line change
Expand Up @@ -2788,7 +2788,7 @@ def entry_graph_block(


def shared_identity_blocks(spec: ExportSpec) -> Dict[str, Any]:
"""The cell-level ck5 identity facts an exported cell must record.
"""The cell-level identity facts an exported cell must record.

``aot_serve.artifact_metadata`` takes ``cell_key`` as a STRING, so the
envelope on its own would carry a stamp WITHOUT the axes the stamp
Expand Down Expand Up @@ -2870,7 +2870,7 @@ class hash folds that entry's ``range_digest`` (the #723 S3 requirement:

CONTRACT-FACTS SHAPE CHANGE (v1 -> v2, pgw#758): this re-keys every
published ``aot-inductor`` cell; single-graph format-1 cells are RETIRED —
correct and expected under ck5 exact identity.
correct and expected under exact identity.

CONTRACT-FACTS SHAPE CHANGE (v2 -> v3, pgw#817): ``shell_digest`` joined
the facts for the (since-retired) regional kind. pgw#846 retires regional
Expand Down
2 changes: 1 addition & 1 deletion src/gen_worker/aot_package.py
Original file line number Diff line number Diff line change
Expand Up @@ -696,7 +696,7 @@ def input_contract(
# a serve-side bind reading `position` would fetch the wrong one
# (measured — it is how pgw#994's `t` went missing). Every cell
# published before pgw#994 has no containers at all, so every row
# is derivable and no live artifact's metadata (or ck6 key) moves
# is derivable and no live artifact's metadata (or cell key) moves
# — see `aot_serve.range_digest`.
row["param"] = leaf.param
row["param_position"] = leaf.param_position
Expand Down
18 changes: 9 additions & 9 deletions src/gen_worker/aot_serve.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@
AOTI_ALIGNMENT = 16
#: Format 2 = multi-graph cells (pgw#758): the envelope carries an
#: ``entries`` map instead of one flat contract. Format-1 cells are RETIRED
#: (ck5 exact identity: a recipe change strands old cells, which is fine).
#: (exact identity: a recipe change strands old cells, which is fine).
ARTIFACT_FORMAT = 2
#: Separator between the entry name and the constant FQN in
#: ``constants.safetensors`` keys. Entry names never contain it (targets are
Expand Down Expand Up @@ -240,8 +240,8 @@ def flavor_label(sku: str, version: str, precision: str) -> str:
return f"aot-{sku}-torch{mm}-{precision}"


# Stamped ck5 cell keys this process LEARNED name aot-inductor artifacts
# (pgw#722 F1 discovery). Published AOT cells ride the ck5 key space as
# Stamped cell keys this process LEARNED name aot-inductor artifacts
# (pgw#722 F1 discovery). Published AOT cells ride the same key space as
# their store flavor — indistinguishable from a dynamo cell's flavor by
# string shape alone — so discovery registers each learned key here and
# :func:`is_aot_ref` consults the set. Without this the executor's kind
Expand All @@ -252,7 +252,7 @@ def flavor_label(sku: str, version: str, precision: str) -> str:


def note_aot_key(cell_key: str) -> None:
"""Record that ``cell_key`` (a stamped ck5 digest) is an AOT cell."""
"""Record that ``cell_key`` (a stamped cell-key digest) is an AOT cell."""
key = str(cell_key or "").strip()
if not key:
return
Expand All @@ -264,7 +264,7 @@ def is_aot_ref(ref: str, family: str = "") -> bool:
"""True when ``ref`` names an AOTI cell (optionally of one family).

Recognizes both the label form (``#aot-<sku>-...``) and any stamped
ck5 key this process learned via :func:`note_aot_key`.
cell key this process learned via :func:`note_aot_key`.
"""
fam, flavor = parse_cell_ref(ref)
if not fam or (family and fam != family):
Expand Down Expand Up @@ -488,7 +488,7 @@ def constants_from_meta(meta: Mapping[str, Any]) -> Tuple[ConstantSpec, ...]:
def range_digest(meta: Mapping[str, Any]) -> str:
"""Canonical digest of the DECLARED admissible traffic of one artifact.

Owed to the ck6 identity lane (pgw#716/#717): declared dim ranges live
Owed to the exact-identity lane (pgw#716/#717): declared dim ranges live
in ``ep.range_constraints``, NOT in the graph nodes — three exports
differing only in declared range produced the identical node-only
digest. A node-only ``graph_hashes`` therefore collides artifacts that
Expand Down Expand Up @@ -527,7 +527,7 @@ def _row(s: InputContract) -> Dict[str, Any]:
}
# pgw#790: the NEGATIVE half of the declared admissible traffic. Two
# classes that differ only in what they REFUSE admit different traffic,
# so the digest must see it or the ck6 collision this function exists to
# so the digest must see it or the collision this function exists to
# close reopens for adapter forks. Keyed only when non-empty: a contract
# that excludes nothing is the contract every already-published cell
# declares, and re-keying the fleet's 144 live checkpoints to add a field
Expand Down Expand Up @@ -566,7 +566,7 @@ def class_hash(


def combined_graph_hash(hashes: Iterable[str]) -> str:
"""The ck6 combined hash, VERBATIM per pgw#716: the first 16 hex chars
"""The combined hash, VERBATIM per pgw#716: the first 16 hex chars
of the sha256 over the newline-joined SORTED per-class hash values
(sorted by the hash string itself, single ``\\n`` joins, no trailing
newline, UTF-8 bytes)."""
Expand Down Expand Up @@ -624,7 +624,7 @@ def artifact_metadata(
into two interpretations of the same bytes. Each entry block carries
``target``/``fork``/``class_dims``/``inputs``/``symbols``/``constants``
(+ ``graph``); this function validates every one, stamps its
``range_digest`` and ``class_hash``, and stamps the ck6
``range_digest`` and ``class_hash``, and stamps the
``combined_graph_hash`` over the sorted per-class hashes. A malformed
contract must fail at MINT, on the pod, not at serve time on a paying
request.
Expand Down
39 changes: 22 additions & 17 deletions src/gen_worker/aot_wrapper_split.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
from typing import Callable, List, Optional, Sequence, Tuple

from . import host_isa
from . import aot_run_impl_split as v2
from . import aot_run_impl_split as runimpl

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -298,11 +298,11 @@ def _reinline(source: str) -> str:

#: Env kill-switch. Not a sealed config knob and not an inductor config —
#: see :func:`install` for why that distinction is what keeps cell identity
#: untouched. Kills v1 and v2 together.
#: untouched. Kills the ctor and run_impl levers together.
DISABLE_ENV = "GEN_WORKER_AOT_WRAPPER_SPLIT_OFF"

#: Kill switch for the pgw#811 ``run_impl`` split alone, so the (much older,
#: much better travelled) v1 ctor split can stay on if v2 ever has to go off.
#: much better travelled) ctor split can stay on if run_impl ever goes off.
DISABLE_V2_ENV = "GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF"

#: How many of the K+1 part compiles may run at once. pgw#809's pool owns
Expand Down Expand Up @@ -340,19 +340,24 @@ def host_compile_jobs(tus: int) -> int:
return max(1, min(tus, budget))


def _emit(outcome: SplitOutcome, version: str = "") -> None:
"""One typed row per wrapper compile. ``version`` distinguishes the
pgw#811 ``run_impl`` split (``v2_applied`` / ``v2_declined``) from v1's
original ``applied`` / ``declined``, so a fleet can tell which of the two
levers fired on any given mint."""
#: Phase-label prefix for the pgw#811 run_impl lever (pgw#958: a name,
#: never a version number).
_RUNIMPL = "runimpl"


def _emit(outcome: SplitOutcome, lever: str = "") -> None:
"""One typed row per wrapper compile. ``lever`` names WHICH split fired:
the pgw#811 ``run_impl`` split (``runimpl_applied`` / ``runimpl_declined``)
versus the original ctor split (bare ``applied`` / ``declined``). A phase
label, deliberately non-numeric — it is not a version counter."""
try:
from . import activity as activity_mod

state = "applied" if outcome.applied else "declined"
activity_mod.emit_event(
EVENT,
outcome.detail(),
phase=f"{version}_{state}" if version else state,
phase=f"{lever}_{state}" if lever else state,
)
except Exception: # pragma: no cover - telemetry must never break a mint
logger.debug("aot-wrapper-split: activity emit failed", exc_info=True)
Expand Down Expand Up @@ -408,7 +413,7 @@ def transform_command(cmd_line: str) -> Tuple[str, Optional[SplitOutcome]]:


# ---------------------------------------------------------------------------
# v2: the run_impl K-way split, driven over K+1 real compiles
# run_impl: the K-way split, driven over K+1 real compiles
# ---------------------------------------------------------------------------

def _object_arg(argv: Sequence[str]) -> Optional[int]:
Expand Down Expand Up @@ -455,9 +460,9 @@ def split_and_compile(cmd_line: str, cwd: str,
text = source.read_text()
except OSError as exc:
_emit(SplitOutcome(False, f"unreadable source: {exc}",
source=str(source)), version="v2")
source=str(source)), lever=_RUNIMPL)
return None
split = v2.split_run_impl(text)
split = runimpl.split_run_impl(text)
if not split.applied:
_emit(SplitOutcome(False, split.reason, source=str(source)))
return None
Expand Down Expand Up @@ -562,13 +567,13 @@ def _patched(cmd_line: str, cwd: str) -> None:
return
_emit(outcome)

# v2 (pgw#811) splits run_impl out of whatever source v1 left behind
# pgw#811 splits run_impl out of whatever source the ctor split left
# — inductor's own when v1 declined, v1's regrouped one when it did
# not. It subsumes the single compile: when it fires it drives every
# compile itself and there is no monolith left to run.
if not os.environ.get(DISABLE_V2_ENV, "").strip():
try:
v2_outcome = split_and_compile(new_cmd, cwd, original)
runimpl_outcome = split_and_compile(new_cmd, cwd, original)
except Exception as exc:
_emit(
SplitOutcome(
Expand All @@ -577,14 +582,14 @@ def _patched(cmd_line: str, cwd: str) -> None:
f"{type(exc).__name__}: {exc}",
source=outcome.source,
),
version="v2",
lever=_RUNIMPL,
)
logger.warning(
"aot-wrapper-split: pgw#811 split failed (%s); recompiling "
"the whole wrapper", type(exc).__name__, exc_info=True)
else:
if v2_outcome is not None:
_emit(v2_outcome, version="v2")
if runimpl_outcome is not None:
_emit(runimpl_outcome, lever=_RUNIMPL)
return

if not outcome.applied:
Expand Down
22 changes: 12 additions & 10 deletions src/gen_worker/cell_key.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
* cozy-local's self-mint (gw#555) looks up / saves its store by the same
key.

The key is the RECIPE DIGEST (ck5, Paul's exact-identity ruling): "look at
The key is the RECIPE DIGEST (Paul's exact-identity ruling): "look at
our code and say 'this is the graph we need' — that is our unique
identifier. If the recipe changes it gets a new identifier, stranding the
old ones, which is fine." No version comparison, no relaxable axes, no
Expand All @@ -36,7 +36,7 @@
torch/triton/cuda/diffusers/transformers VERSION axes —
content, never version strings

ck6 (pgw#990) DROPS ``code_closure`` from the key. It is still RECORDED on
pgw#990 DROPS ``code_closure`` from the key. It is still RECORDED on
every artifact and still drives ``compile_cache``'s local re-trace memo, but
it is not identity: Paul's final ruling is that identity is the COMPUTATION
(traced graph x sm x toolchain x env_seal) and "code hashes are a memo, never
Expand All @@ -49,7 +49,7 @@
and runtime compat checks (``compile_cache.verify``) only: ``sku`` (pgw#691
— no guard or artifact fact observes it), ``cuda_driver`` (gw#577),
``torch``/``triton``/``cuda``/``gen_worker``/``diffusers``/``transformers``
version strings and ``image_digest`` (ck5 — their CONTENT rides the
version strings and ``image_digest`` (exact identity — their CONTENT rides the
toolchain and code_closure axes; version strings and image identity are
observability, not identity).

Expand All @@ -68,12 +68,14 @@
from typing import Any, Dict, Mapping, Tuple
from . import env_seal

# ck2 -> ck3 (pgw#691): sku left. ck3 -> ck4 (pgw#696): env_seal joined.
# ck4 -> ck5 (Paul's exact-identity ruling): the key became the RECIPE
# digest — toolchain + code_closure content joined; every version axis
# left. ck5 is FINAL: new identity facts ride the content digests (seal_v /
# closure/toolchain values), never new axes.
KEY_SCHEME = "ck6"
# pgw#958 (DESIGN-RULINGS §1.27(g); Paul 2026-08-04, reaffirmed 2026-08-07
# over pgw#990's ck6 — "the ck6 is wrong and can be removed, the v1 is the
# correct one"): the counter restarts at 1, and the pre-existing ck1..ck6
# corpus is PURGED in the same cut (th#1636), so no two cells ever share a
# scheme token with different meanings. ck1 is the only live scheme; new
# identity facts ride the content digests (seal_v / closure / toolchain
# values), never new axes and never a new scheme number.
KEY_SCHEME = "ck1"
_PREFIX = KEY_SCHEME + "-"
# The key digest doubles as the store flavor token, whose shared grammar
# (th#597 C5: [a-z0-9][a-z0-9._-]{0,63}, Go+Py identical) caps tokens at 64
Expand Down Expand Up @@ -243,7 +245,7 @@ def from_artifact_metadata(meta: Mapping[str, Any]) -> CellKey:
store. A cell with no computable key is refused and re-minted.

EXPORTED (``aot-inductor``) cells are refused here BY NAME (pgw#735): they
ride the same ck5 key space — the axis names are what :func:`from_axes`
ride the same key space — the axis names are what :func:`from_axes`
validates, and the kind is an envelope value, so no scheme bump was needed —
but their axes are not an inductor cache's, and their key is STAMPED at mint.
Read ``meta["cell_key"]``; do not recompute an exported cell's identity from
Expand Down
Loading
Loading