diff --git a/changelog.d/pgw995.md b/changelog.d/pgw995.md new file mode 100644 index 00000000..acaf6bb1 --- /dev/null +++ b/changelog.d/pgw995.md @@ -0,0 +1,71 @@ +- **pgw#995: env carries values, never a branch selection — wave 2, plus the + BEHAVIOUR axis the first census could not see.** pgw#990 deleted + `GEN_WORKER_PREFER_AOT`, the flag that gated both the mint recipe and cell + discovery, silently disarmed on a release rebuild and cost three pod attempts. + This is the class rather than the instance. The pgw#929/931 census is real — + 79 classified (file, variable) pairs — but its six classifications all answer + **WHERE** a read happens relative to the config pipeline, never **WHETHER** it + selects behaviour, and the postmortem lives entirely on the second axis. The + evidence it was missing rather than implicit: `GEN_WORKER_AOT_EXPORT_PARALLEL` + and `_REUSE` were filed as **LIBRARY** (torch has never heard of either name — + they are first-party feature gates), and `GEN_WORKER_BG_YIELD`, + `_EAGER_FIRST_BOOT` and `_MINT_IN_PROCESS` as **STANDALONE** ("a CLI that loads + no app config") while all three were read from `executor.py` on the serving hot + path of every production pod. + **Three switches DELETED**, each default-ON with zero fleet declarations, so + deleting the env made the shape every pod already ran unconditional and no + pod's behaviour changed: `GEN_WORKER_BG_YIELD` (pgw#677 bg-yield, and the + legacy idle-gate tree it selected), `GEN_WORKER_EAGER_FIRST_BOOT` (pgw#671 + eager-first, and `mint_delegate.REFUSAL_EAGER_FIRST_DISABLED` with it — a + refusal reason that can no longer be returned is a cause a reader hunts for and + never finds), and `GEN_WORKER_AOT_WRAPPER_SPLIT_OFF`. + **Four gates KEPT, each with a stated reason, because "delete every kill + switch" is wrong in two different directions.** `GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF` + looks identical to its deleted sibling in the same file and is **LIVE** — 5 + SDXL releases declare it, 1 endpoint carries a non-deleted entry — so zero + declarations is a fact you MEASURE, not one you read off the code. + `GEN_WORKER_AOT_EXPORT_PARALLEL` / `_REUSE` are default-OFF and dark by their + own docstrings: deleting those gates would make an unproven path + unconditional, which is a **launch, not a deletion**. `GEN_WORKER_HOST_MOVE_GUARD` + and the pgw#980 probe pair are ruled exceptions with named threats. + **The tests stop using env as a red-verification seam.** `BG_YIELD=0` and + `EAGER_FIRST_BOOT=0` drove RED arms in four files; an env kept alive to be a + test seam is still an env a release rebuild can flip. Those arms are replaced + by absolute-bound assertions plus source-level guards proving the deleted arm + is **unreachable**, not merely unused — deleting a switch and deleting its + tests looks identical to deleting a switch and leaving a second reader behind, + which is exactly how PREFER_AOT kept a live gate after one was believed gone. + **`scripts/lint_config_reads.py` now enforces the behaviour axis.** An AST pass + flags any env read that feeds a conditional and fails the build unless the + (file, variable) pair is in `BEHAVIOUR_GATES` **with the threat it defends + against** — "it is useful" and "it is off by default" are rejected. Stale + exemptions fail too. Structurally the same bar as the `mint_recipe` guard + shipped in 0.93.2. + **`GEN_WORKER_MINT_IN_PROCESS` is NOT deleted, and it is the one remaining + defect.** Default-ON, zero declarations, and `enable_compiled(delegate=False)` + is a strictly better parameter seam that already exists — blocked only on ten + test sites across seven files that force the shape via the env and drive the + executor. Recorded in `BEHAVIOUR_GATES` as a burn-down, not an exception. + +- **pgw#995 deliverable 2: the local rig can finally see hub env delivery + (`micro_mint_rig.py --hub-env`).** The rig runs the whole mint machinery on + this box, which is why a change can be proven before PyPI — but it + **constructed its own environment** (`mint_process.child_env` for the mint + child, `dict(os.environ)` for the adopting process), shapes no production pod + ever has. So the chain that actually delivers env to a pod — worker function + declares → `release_env_declarations` → `endpoint_env_entries` → + `EndpointEnvService.Resolve` → pod env → `Settings` — was invisible to every + test in this repo, and the regression class that took PREFER_AOT dark had + exactly one detector: a pod. `tests/harness/hub_env.py` models the hub's + resolution rule (an entry reaches the pod only if the release DECLARES its + name, plus the reserved-namespace defence) and reports withholdings in + th#1650's typed vocabulary. `--hub-env` boots the mint child through it, + **stripping ambient values** so a developer's own shell cannot stand in for a + hub-delivered one — that substitution is what the blind spot was made of, and a + mode that allowed it would be decoration. `tests/test_hub_env_delivery_pgw995.py` + drives the real `config.load_settings()` over hub-resolved environments and + reproduces the postmortem in milliseconds: release N declares the name and the + value reaches `Settings`; release N+1 is rebuilt without the declaration, + nobody touches the entry, and the value is withheld **and said so**. Full + HelloAck-shaped boot (Vault, `applies_to` version/tag matching, the mTLS + resolve path) is filed on pgw#995 with an owner. Hub half: **th#1650**. diff --git a/scripts/config_reads_allowlist.txt b/scripts/config_reads_allowlist.txt index 218bbdfc..0bd8e464 100644 --- a/scripts/config_reads_allowlist.txt +++ b/scripts/config_reads_allowlist.txt @@ -139,10 +139,7 @@ src/gen_worker/models/svdq.py::GEN_WORKER_SVDQ_ENGINE STANDALONE GEN_WORKER_SVDQ src/gen_worker/aot_resume.py::GEN_WORKER_MINT_RESUME_MAX_BYTES STANDALONE GEN_WORKER_MINT_RESUME_MAX_BYTES, resume-area capacity bound. src/gen_worker/aot_resume.py::GEN_WORKER_MINT_RESUME_DIR STANDALONE GEN_WORKER_MINT_RESUME_DIR. pgw#929: the ENABLE used to be hidden inside an empty string; `resume_enabled()` now states it. src/gen_worker/mint_delegate.py::GEN_WORKER_MINT_IN_PROCESS STANDALONE GEN_WORKER_MINT_IN_PROCESS forces the in-process mint shape, which VIOLATES the liveness contract and is kept reachable to prove that, not to run. -src/gen_worker/mint_delegate.py::GEN_WORKER_EAGER_FIRST_BOOT STANDALONE GEN_WORKER_EAGER_FIRST_BOOT, second reader; the two switches move together. -src/gen_worker/executor.py::GEN_WORKER_BG_YIELD STANDALONE GEN_WORKER_BG_YIELD=0 restores the pre-pgw#677 shape. See the kill-switch note below. -src/gen_worker/executor.py::GEN_WORKER_EAGER_FIRST_BOOT STANDALONE GEN_WORKER_EAGER_FIRST_BOOT=0. See the kill-switch note below. # --------------------------------------------------------------------------- # TRIPWIRE — guards whose entire purpose is to fire on a misconfiguration. # --------------------------------------------------------------------------- @@ -150,42 +147,65 @@ src/gen_worker/content_credentials.py::$env_name TRIPWIRE GEN_WORKER_C2PA_KEY_PE src/gen_worker/models/memory.py::GEN_WORKER_FORBID_CPU_OFFLOAD TRIPWIRE GEN_WORKER_FORBID_CPU_OFFLOAD, now enforced at the REAL placement boundary (pgw#929 AMBIGUOUS #1). Read as env rather than Settings because a control-plane box exports it box-wide with no worker config in sight. src/gen_worker/benchmarks/swap_latency.py::GEN_WORKER_FORBID_CPU_OFFLOAD TRIPWIRE GEN_WORKER_FORBID_CPU_OFFLOAD, the original single reader. Retained: refusing the benchmark is still correct. src/gen_worker/host_move_guard.py::GEN_WORKER_HOST_MOVE_GUARD TRIPWIRE GEN_WORKER_HOST_MOVE_GUARD, the pgw#763 host-move guard's disable escape hatch. See the kill-switch note below. -src/gen_worker/aot_wrapper_split.py::GEN_WORKER_AOT_WRAPPER_SPLIT_OFF TRIPWIRE GEN_WORKER_AOT_WRAPPER_SPLIT_OFF, v1 ctor-split disable. See the kill-switch note below. src/gen_worker/aot_mint.py::PODGUARD_STATE TRIPWIRE External watchdog adapter with a producer OUTSIDE this process (`podguard.arm()` on rented pods), so there is no argv to move it to and no Settings that could own it. Gated on podguard_status()==armed; pgw#929 owns its validation and observability permanently and deletion is explicitly out of scope. # --------------------------------------------------------------------------- -# NOTE — the seven "kill switches" pgw#929 lists for deletion. FIVE can go once -# their unconditional path has a named observable; TWO CANNOT, and saying so is -# part of the finding. +# NOTE — the BEHAVIOUR axis (pgw#995), and the kill-switch verdicts. # -# Measured on the standing hub 2026-08-03: `release_env_declarations` and -# `endpoint_env_entries` carry exactly two GEN_WORKER_* names fleet-wide, -# GEN_WORKER_PREFER_AOT (19 declarations, 2 live entries) and -# GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF (5 declarations, 1 live entry). None of the -# seven kill switches is declared anywhere. +# The six classifications above answer WHERE a read happens relative to the +# config pipeline. They do NOT answer whether the read SELECTS BEHAVIOUR, and +# those are orthogonal. `GEN_WORKER_PREFER_AOT` was a behaviour switch that +# silently disarmed on a release rebuild and took the whole AOT path dark for +# three pod attempts; nothing in a WHERE-shaped classification could have +# flagged it. Evidence the axis was missing rather than implicit: this file used +# to classify `GEN_WORKER_AOT_EXPORT_PARALLEL` and `_REUSE` as LIBRARY (torch has +# never heard of either name — they are first-party feature gates) and +# `GEN_WORKER_BG_YIELD` / `_EAGER_FIRST_BOOT` / `_MINT_IN_PROCESS` as STANDALONE +# ("a CLI that loads no app config") while all three were read from executor.py +# on the serving hot path of every production pod. # -# Zero declarations is NOT the same fact as safe-to-delete, and the direction of -# the default decides which: +# So: `scripts/lint_config_reads.py` now also asserts the behaviour axis, and +# Paul's rule is enforced rather than restated — env carries CONFIG, SECRETS and +# TUNING VALUES. A branch selector needs typed config, a loud typed observable, +# and a named threat. # -# DEFAULT-ON (HOST_MOVE_GUARD, MINT_IN_PROCESS, EAGER_FIRST_BOOT, BG_YIELD, -# AOT_WRAPPER_SPLIT_OFF) — deleting the env makes the CURRENT -# PRODUCTION BEHAVIOUR unconditional. Safe. Blocked only on the -# replacement observable pgw#929 requires, plus the fact that the -# tests set BG_YIELD=0 / EAGER_FIRST_BOOT=0 as red-verification -# seams: those need a real parameter, not an env, before the -# switch goes. +# THE DIRECTION OF THE DEFAULT DECIDES WHETHER A GATE CAN BE DELETED: # -# DEFAULT-OFF (AOT_EXPORT_PARALLEL, AOT_EXPORT_REUSE) — deleting the env would -# make an UNPROVEN path unconditional. Both ship dark by their own -# docstrings ("turning it on needs the export-phase VRAM -# measurement first"; "ships dark until a real family's mint has -# run the gate on a pod"), and neither enable path has ever run in -# production. Making them unconditional is not a deletion, it is a -# launch. pgw#929's "delete seven kill switches" is WRONG for -# these two. +# DEFAULT-ON — deleting the env makes the CURRENT PRODUCTION BEHAVIOUR +# unconditional, so no pod changes behaviour. Safe. +# DEFAULT-OFF — deleting the env makes an UNPROVEN path unconditional. That +# is not a deletion, it is a LAUNCH. pgw#929's "delete seven kill +# switches" is WRONG for these, and saying so is the finding. # -# If they encode a decision nobody makes — which the measurement -# supports — the §2.2 answer is to delete the FEATURES, ~500 lines -# of dark code. That is a call for Paul, not a side effect of an -# env census, so it is recorded here rather than taken. +# ...but only together with a MEASURED declaration count. Zero fleet +# declarations is what licenses a deletion, and it is a fact you query, not one +# you infer from the code — `GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF` looks identical +# to its deleted sibling in this file and is LIVE on the standing hub. +# +# VERDICTS (declaration counts measured on the standing hub 2026-08-03): +# +# DELETED by pgw#995 — default-ON, zero declarations: +# GEN_WORKER_BG_YIELD pgw#677 bg-yield is now unconditional +# GEN_WORKER_EAGER_FIRST_BOOT pgw#671 eager-first is now unconditional +# GEN_WORKER_AOT_WRAPPER_SPLIT_OFF v1 ctor-split is now unconditional +# +# NOT DELETED, and each for a stated reason: +# GEN_WORKER_MINT_IN_PROCESS default-ON, zero declarations, SAFE to +# delete — blocked only on ten test sites across seven files that +# force the shape via this env and drive the executor. A strictly +# better parameter seam already exists (`enable_compiled(delegate= +# False)` -> `caller_forced_in_process`). Owner on pgw#995. +# GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF LIVE: 5 SDXL releases declare it, 1 +# endpoint carries a non-deleted entry. Deleting it changes a +# running endpoint. +# GEN_WORKER_AOT_EXPORT_PARALLEL default-OFF and dark by its own +# GEN_WORKER_AOT_EXPORT_REUSE docstring. Deleting the gate LAUNCHES an +# unproven path. If they encode a decision nobody makes, the §2.2 +# answer is to delete the ~500 lines of FEATURE — a call for Paul, +# not a side effect of an env census. +# GEN_WORKER_HOST_MOVE_GUARD ruled exception: safety guard, on by +# default, named threat, documented. Do not touch. +# GEN_WORKER_PROBE / _PUBLISH_ARMED security boundary; deliberately NOT a +# Settings field so tenant-adjacent code cannot reach it. +# PODGUARD_STATE producer is outside this process. # --------------------------------------------------------------------------- diff --git a/scripts/lint_config_reads.py b/scripts/lint_config_reads.py index bec74c95..a81de34f 100644 --- a/scripts/lint_config_reads.py +++ b/scripts/lint_config_reads.py @@ -221,6 +221,207 @@ def load_allowlist() -> Tuple[Dict[Tuple[str, str], str], List[str]]: return allowed, errors +# --------------------------------------------------------------------------- +# pgw#995 — the BEHAVIOUR axis +# --------------------------------------------------------------------------- +# +# The six classifications above answer WHERE a read happens relative to the +# config pipeline. None of them answers whether the read SELECTS BEHAVIOUR, and +# those are orthogonal questions. `GEN_WORKER_PREFER_AOT` was a behaviour switch +# that silently disarmed on a release rebuild and took the entire AOT path dark +# for three pod attempts; nothing WHERE-shaped could have flagged it. The +# allowlist proves the axis was missing rather than implicit: it classified +# `GEN_WORKER_AOT_EXPORT_PARALLEL`/`_REUSE` as LIBRARY (torch has never heard of +# either name) and three serving-hot-path switches as STANDALONE ("a CLI that +# loads no app config"). +# +# Paul's rule: env carries CONFIG, SECRETS and TUNING VALUES. A branch selector +# needs typed config, a loud typed observable, and a named threat. This gate +# makes a NEW one fail the build instead of being noticed three pods later. + +#: (path, ENV_NAME) -> the named threat this gate defends against. +#: A gate lives here ONLY with a threat a reader can evaluate. "It is useful" +#: and "it is off by default" are not threats. +BEHAVIOUR_GATES: Dict[Tuple[str, str], str] = { + ("src/gen_worker/host_move_guard.py", "GEN_WORKER_HOST_MOVE_GUARD"): + "RULED EXCEPTION. Safety guard, ON by default, disabled only with =0. " + "Threat: a silent host-RAM offload turns a serving pod into a " + "swap-thrashing one that still answers health checks. Documented in " + "CLAUDE.md; explicitly out of scope for every env sweep.", + ("src/gen_worker/procsplit/actions.py", "GEN_WORKER_PROBE"): + "SECURITY BOUNDARY (pgw#980). Marks the pod a live-edit probe and " + "DISARMS cell publish in the parent's action allowlist. Threat: a probe " + "pod publishing a cell minted from hand-edited source into the fleet " + "store. Deliberately NOT a Settings field: `authorize` is the boundary, " + "and a guard that depends on a config load having succeeded has an " + "unarmed window.", + ("src/gen_worker/procsplit/actions.py", "GEN_WORKER_PROBE_PUBLISH_ARMED"): + "SECURITY BOUNDARY (pgw#980). The separate second decision that re-arms " + "publish on a marked probe. Two names so 'this is a probe' and 'this " + "probe may write' can never be satisfied by one value.", + ("src/gen_worker/procsplit/__init__.py", "GEN_WORKER_COMPUTE_CHILD"): + "BOOTSTRAP. Decides WHICH OF TWO PROGRAMS this process is, before " + "_run_main and before any config can exist. Threat: none — it is not a " + "policy, it is the process's own identity, and there is no earlier " + "carrier than the environment it was exec'd with.", + ("src/gen_worker/supervisor.py", "GEN_WORKER_SUPERVISOR"): + "BOOTSTRAP. Pre-fork supervisor predicate, runs before the process " + "entry. Same identity-not-policy reasoning.", + ("src/gen_worker/supervisor.py", "GEN_WORKER_SUPERVISED"): + "BOOTSTRAP. Pre-fork re-entry guard; without it the supervisor forks " + "itself forever.", + ("src/gen_worker/models/memory.py", "GEN_WORKER_FORBID_CPU_OFFLOAD"): + "TRIPWIRE at the real placement boundary. Read as env rather than " + "Settings because a control-plane box exports it box-wide with no worker " + "config in sight. Threat: a CPU-offloading run on the shared dev box.", + ("src/gen_worker/benchmarks/swap_latency.py", "GEN_WORKER_FORBID_CPU_OFFLOAD"): + "TRIPWIRE, the original single reader. Refusing the benchmark is still " + "correct. Same threat, same box-wide-export reasoning.", + ("src/gen_worker/mint_delegate.py", "GEN_WORKER_MINT_IN_PROCESS"): + "DEFECT, NOT AN EXCEPTION — listed so the gate is green while it is " + "burned down, exactly as VIOLATION lines are. Default-ON, zero fleet " + "declarations, and `enable_compiled(delegate=False)` is a strictly " + "better parameter seam that already exists. Blocked ONLY on ten test " + "sites across seven files that force the shape via this env and drive " + "the executor. Owner and scope on pgw#995. Do not add a second reader.", + ("src/gen_worker/aot_export_parallel.py", "GEN_WORKER_AOT_EXPORT_PARALLEL"): + "DARK FEATURE, default OFF. NOT deletable: deleting the gate would make " + "an unproven path unconditional, which is a LAUNCH, not a deletion. " + "Threat: an unmeasured export-phase VRAM footprint OOMs a 74-minute " + "phase. `decide()` emits the decision on EVERY mint whether or not the " + "flag is on, so the gate's state is observable rather than silent — " + "which is the property PREFER_AOT lacked.", + ("src/gen_worker/aot_export_reuse.py", "GEN_WORKER_AOT_EXPORT_REUSE"): + "DARK FEATURE, default OFF. Same launch-not-deletion reasoning. Threat: " + "a reused export whose artifact is not byte-identical silently changes " + "a compiled program. The gate byte-compares every emitted file and " + "falls back on any doubt (ReuseUnproven).", + ("src/gen_worker/aot_wrapper_split.py", "GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF"): + "LIVE ON THE FLEET, and that is why it survives its deleted sibling: 5 " + "SDXL releases declare it and 1 endpoint carries a non-deleted entry " + "(standing hub, 2026-08-03). Deleting a live switch changes a running " + "endpoint. Threat: the pgw#811 run_impl split regressing a family, with " + "no way to unstick it short of a release.", + ("src/gen_worker/lifecycle.py", "$ENV_VAR"): + "READ-ONLY WARNING PREDICATE. Re-reads the hub-delivered topology only " + "to decide whether the 'GPUs are invisible' warning applies. Selects a " + "log line, never a code path.", + ("src/gen_worker/executor.py", "RUNPOD_POD_ID"): + "DEFECT, listed to keep the gate green while it burns down. A vendor env " + "used as a proxy for 'managed runtime'. Blocked on pgw#921/th#1488 " + "RuntimeIdentity.managed; pgw#929 AMBIGUOUS #5 forbids papering over it " + "with a vendor Settings field.", + ("src/gen_worker/executor.py", "RUNPOD_PROVIDER"): + "DEFECT, same site and same blocker as RUNPOD_POD_ID above.", + ("src/gen_worker/content_credentials.py", "$env_name"): + "TRIPWIRE that REFUSES THE BOOT (th#1307). Threat: a C2PA private key " + "reaching a pod. Carries no behaviour of its own — its only outcome is " + "a loud refusal.", + +} + +#: Predicate-shaped function names: a `return ` inside one of these is +#: a behaviour selection even without a syntactic `if`. +_PREDICATE_SUFFIXES = ("enabled", "disabled", "_on", "_off", "armed", "forced") + + +class BehaviourVisitor(ast.NodeVisitor): + """Env reads whose value reaches a CONDITIONAL rather than a value slot. + + Deliberately syntactic and conservative. It cannot follow a read through a + variable into an `if` three functions away, and it does not try — a gate + that pretends to completeness it does not have is worse than one whose + reach is stated. What it DOES catch is every shape the four switches + deleted by pgw#995 were written in, which is the shape this defect keeps + being written in. + """ + + def __init__(self) -> None: + self.hits: List[Tuple[int, str]] = [] + + def _collect(self, node: ast.AST) -> None: + sub = EnvVisitor() + sub.visit(node) + self.hits.extend(sub.hits) + + def _scan_conditions(self, tree: ast.AST, consts: Dict[str, str]) -> None: + for node in ast.walk(tree): + if isinstance(node, (ast.If, ast.While, ast.IfExp, ast.Assert)): + self._collect_with(node.test, consts) + elif isinstance(node, ast.comprehension): + for cond in node.ifs: + self._collect_with(cond, consts) + elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + returns_bool = ( + isinstance(node.returns, ast.Name) + and node.returns.id == "bool") + predicate_name = node.name.lower().endswith(_PREDICATE_SUFFIXES) + if not (returns_bool or predicate_name): + continue + for inner in ast.walk(node): + if isinstance(inner, ast.Return) and inner.value is not None: + self._collect_with(inner.value, consts) + + def _collect_with(self, node: ast.AST, consts: Dict[str, str]) -> None: + sub = EnvVisitor() + sub.consts = dict(consts) + sub.visit(node) + self.hits.extend(sub.hits) + + +def scan_behaviour() -> Dict[Tuple[str, str], int]: + """Every env read outside `config/` that feeds a conditional.""" + sites: Dict[Tuple[str, str], int] = {} + for path in sorted(SRC_ROOT.rglob("*.py")): + if CONFIG_PKG in path.parents or path == CONFIG_PKG: + continue + try: + tree = ast.parse(path.read_text(encoding="utf-8")) + except SyntaxError: # pragma: no cover - the other pass reports it + continue + consts = EnvVisitor() + consts.load_consts(tree) + visitor = BehaviourVisitor() + visitor._scan_conditions(tree, consts.consts) + rel = str(path.relative_to(REPO)) + for lineno, name in visitor.hits: + sites.setdefault((rel, name or UNRESOLVED), lineno) + return sites + + +def check_behaviour_gates() -> List[str]: + """Paul's rule, enforced: env carries values, never a branch selection.""" + found = scan_behaviour() + errors: List[str] = [] + for key in sorted(set(found) - set(BEHAVIOUR_GATES)): + path, name = key + errors.append( + f"{path}:{found[key]} reads {name} from the environment and feeds it " + f"to a CONDITIONAL. Env vars are for CONFIG and SECRETS, never logic " + f"or behaviour switches (Paul, standing rule). GEN_WORKER_PREFER_AOT " + f"was exactly this: it gated the mint recipe and cell discovery, " + f"silently disarmed on a release rebuild, and cost three pod attempts " + f"before anyone noticed the AOT path was dark.\n" + f" Fix it: make the branch unconditional (if its default is ON and " + f"nothing declares it, deleting the gate changes NO pod's behaviour), " + f"or move the decision to typed config with a loud typed observable.\n" + f" If it genuinely must stay, add ('{path}', '{name}') to " + f"BEHAVIOUR_GATES in {Path(__file__).name} with the THREAT it defends " + f"against — not 'it is useful' and not 'it is off by default'.") + for key in sorted(set(BEHAVIOUR_GATES) - set(found)): + path, name = key + errors.append( + f"BEHAVIOUR_GATES lists ('{path}', '{name}') but no such conditional " + f"env read exists any more. Delete the entry — a stale exemption is " + f"the second carrier this whole gate exists to prevent (§4.22).") + for key, threat in BEHAVIOUR_GATES.items(): + if len(threat.strip()) < 40: + errors.append( + f"BEHAVIOUR_GATES{key}: the threat must be specific enough for a " + f"reader to evaluate, got {threat!r}") + return errors + + def check_owned_names_known(names: Set[str]) -> List[str]: """Every owned-namespace name read in src/ must be known to the loader.""" sys.path.insert(0, str(REPO / "src")) @@ -246,6 +447,7 @@ def check_owned_names_known(names: Set[str]) -> List[str]: def main() -> int: sites, names = scan() allowed, errors = load_allowlist() + errors.extend(check_behaviour_gates()) for key in sorted(set(sites) - set(allowed)): path, name = key diff --git a/scripts/micro_mint_rig.py b/scripts/micro_mint_rig.py index 594d21fb..749bde6f 100755 --- a/scripts/micro_mint_rig.py +++ b/scripts/micro_mint_rig.py @@ -38,6 +38,24 @@ * `nice`, so a compile cannot starve the box's other agents; * a load gate: refuse to start above 1-min load 24; * `GEN_WORKER_HOST_MOVE_GUARD` untouched — the rig never disables it. + +THE ENV-DELIVERY MODE (pgw#995, `--hub-env`). The rig above builds the child's +environment itself — `mint_process.child_env` plus a few rig keys — which is a +shape no production pod ever has. So the chain that actually delivers env to a +pod was invisible to it, and to every other test in this repo: + + worker function declares env -> release_env_declarations + operator sets a value -> endpoint_env_entries + pod launch -> EndpointEnvService.Resolve -> pod env -> Settings + +That chain is what took `GEN_WORKER_PREFER_AOT` dark: a release rebuild stopped +declaring the name, the hub withheld the live entry SILENTLY, and three pod +attempts went by. `--hub-env` boots the mint child through it instead — the +child's environment is what the hub's rule would have delivered, ambient values +are STRIPPED so a developer's shell cannot stand in for a delivered one, and any +withholding is reported as a rig fact rather than a missing variable nobody +looks for. The model lives in `tests/harness/hub_env.py`; the first regression +tests are `tests/test_hub_env_delivery_pgw995.py`. """ from __future__ import annotations @@ -272,9 +290,39 @@ def _mint_request( task, workdir=workdir, cap_bytes=cap_bytes) +#: What the rig's endpoint function DECLARES, the way a real build reads it off +#: the function schema. Deliberately tiny: the point is the delivery mechanism, +#: not the breadth of the catalogue. +RIG_DECLARED_ENV = ("HF_TOKEN",) + +#: Ambient names the rig REFUSES to let through in `--hub-env` mode. Without +#: this the developer's own shell satisfies the assertion and the mode proves +#: nothing — which is the exact substitution the blind spot was made of. +RIG_STRIPPED_ENV = ("HF_TOKEN",) + + +def hub_delivered_env( + base: Dict[str, str], entries: Optional[Dict[str, str]] = None, +) -> tuple: + """(env, withheld) as the hub would resolve them for this rig's release.""" + sys.path.insert(0, str(REPO / "tests")) + from harness import hub_env as _hub + + delivery = _hub.resolve( + _hub.declared_by(list(RIG_DECLARED_ENV)), + _hub.EndpointEnvEntries(dict(entries or {}))) + env = _hub.pod_environ(base, delivery, strip=RIG_STRIPPED_ENV) + return env, [ + {"name": w.name, "reason": w.reason, "detail": w.detail} + for w in delivery.withheld + ] + + def run_cycle( root: Path, *, stage: str = "all", force_load: bool = False, device: str = "auto", vehicle: str = DEFAULT_VEHICLE, + hub_env_mode: bool = False, + hub_env_entries: Optional[Dict[str, str]] = None, ) -> RigResult: from harness import rig_vehicles @@ -346,6 +394,13 @@ def run_cycle( g0 = time.monotonic() phases: List[str] = [] env = dict(mp.child_env(request)) + hub_withheld: List[Dict[str, str]] = [] + if hub_env_mode: + # pgw#995: the child no longer inherits whatever this process carries. + # It boots with what the HUB would have delivered for this release — + # so a name the release stops declaring disappears here, locally, the + # way it disappeared on the pod nobody was watching. + env, hub_withheld = hub_delivered_env(env, hub_env_entries) env["PYTHONPATH"] = os.pathsep.join( [str(REPO / "tests"), str(REPO / "src"), *veh.syspath, env.get("PYTHONPATH", "")]) @@ -364,6 +419,8 @@ def run_cycle( leg.facts = { "status": outcome.status, "exit_code": outcome.exit_code, + "hub_env_mode": hub_env_mode, + "hub_env_withheld": hub_withheld, "phases_seen": sorted(set(phases)), "phase_seconds": dict(report.phases) if report else {}, "peak_vram_bytes": int(report.peak_vram_bytes) if report else 0, @@ -572,6 +629,15 @@ def main(argv: Optional[List[str]] = None) -> int: help="WHAT to mint: 'tiny' (pgw#978's one-entry plumbing toy) or " "'micro' (pgw#997's org-worker-shaped examples/micro-diffusion — " "3 entries, container inputs, generated weights)") + parser.add_argument( + "--hub-env", action="store_true", + help="boot the mint child through the hub's env-delivery rule " + "(declarations x entries) instead of this process's environment; " + "ambient values are stripped and withholdings are reported") + parser.add_argument( + "--hub-env-entry", action="append", default=[], metavar="NAME=VALUE", + help="an endpoint_env_entries row the operator has set; repeatable. " + "Only names the release DECLARES are delivered.") args = parser.parse_args(list(argv) if argv is not None else None) root = Path(args.root) @@ -580,8 +646,16 @@ def main(argv: Optional[List[str]] = None) -> int: root.mkdir(parents=True, exist_ok=True) try: + entries: Dict[str, str] = {} + for raw in args.hub_env_entry: + name, sep, value = str(raw).partition("=") + if not sep: + parser.error(f"--hub-env-entry expects NAME=VALUE, got {raw!r}") + entries[name.strip()] = value result = run_cycle(root, stage=args.stage, force_load=args.force_load, - device=args.device, vehicle=args.vehicle) + device=args.device, vehicle=args.vehicle, + hub_env_mode=args.hub_env, + hub_env_entries=entries) except RigRefused as exc: print(f"REFUSED: {exc}", file=sys.stderr) return 2 diff --git a/src/gen_worker/aot_wrapper_split.py b/src/gen_worker/aot_wrapper_split.py index 80d23488..cd39334b 100644 --- a/src/gen_worker/aot_wrapper_split.py +++ b/src/gen_worker/aot_wrapper_split.py @@ -296,13 +296,22 @@ def _reinline(source: str) -> str: # Mint-path installation # --------------------------------------------------------------------------- -#: Env kill-switch. Not a sealed config knob and not an inductor config — -#: see :func:`install` for why that distinction is what keeps cell identity -#: untouched. Kills the ctor and run_impl levers together. -DISABLE_ENV = "GEN_WORKER_AOT_WRAPPER_SPLIT_OFF" +#: pgw#995: ``GEN_WORKER_AOT_WRAPPER_SPLIT_OFF`` is GONE. It defaulted ON, no +#: release ever declared it and no endpoint ever set it, so deleting it makes +#: the shape every pod already ran unconditional. An env that selects which +#: program runs is the class that took `GEN_WORKER_PREFER_AOT` dark for three +#: pod attempts; a switch nobody has ever thrown carries none of the safety it +#: appears to and all of the silent-disarm risk. #: Kill switch for the pgw#811 ``run_impl`` split alone, so the (much older, #: much better travelled) ctor split can stay on if run_impl ever goes off. +#: pgw#995 KEEPS this one, and the asymmetry with its deleted sibling above is +#: the finding rather than an oversight: this name is LIVE — five SDXL releases +#: (0.2.111/112/113/116/117) declare it and one endpoint carries a non-deleted +#: `endpoint_env_entries` row (verified on the standing hub 2026-08-03). Deleting +#: a live switch changes a running endpoint's behaviour; deleting a never-thrown +#: one cannot. "Zero declarations" is the fact that licenses a deletion, and it +#: is a fact you MEASURE, not one you assume from the code. DISABLE_V2_ENV = "GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF" #: How many of the K+1 part compiles may run at once. pgw#809's pool owns @@ -526,14 +535,11 @@ def install() -> bool: ``code_closure`` axis is untouched too. No cell is re-keyed. (Both facts are asserted by tests/test_aot_wrapper_split_pgw793.py.) - Returns True when installed, False when already installed or disabled. + Returns True when installed, False when already installed. """ global _installed if _installed: return False - if os.environ.get(DISABLE_ENV, "").strip(): - logger.info("aot-wrapper-split: disabled by %s", DISABLE_ENV) - return False try: from torch._inductor import cpp_builder except Exception: @@ -632,7 +638,6 @@ def _patched(cmd_line: str, cwd: str) -> None: "CHUNK", "EVENT", "MIN_STATEMENTS", - "DISABLE_ENV", "DISABLE_V2_ENV", "JOBS_ENV", "SERVING_HEADROOM_CPUS", diff --git a/src/gen_worker/config/loader.py b/src/gen_worker/config/loader.py index 8c7910aa..f2d02e3f 100644 --- a/src/gen_worker/config/loader.py +++ b/src/gen_worker/config/loader.py @@ -138,9 +138,6 @@ "GEN_WORKER_INTERNAL_OBJECT_HOSTS", "GEN_WORKER_AOT_EXPORT_PARALLEL", "GEN_WORKER_AOT_EXPORT_REUSE", - "GEN_WORKER_AOT_WRAPPER_SPLIT_OFF", - "GEN_WORKER_BG_YIELD", - "GEN_WORKER_EAGER_FIRST_BOOT", "GEN_WORKER_MINT_IN_PROCESS", "GEN_WORKER_HOST_MOVE_GUARD", "GEN_WORKER_FORBID_CPU_OFFLOAD", diff --git a/src/gen_worker/executor.py b/src/gen_worker/executor.py index f71c3e35..1006b5da 100644 --- a/src/gen_worker/executor.py +++ b/src/gen_worker/executor.py @@ -655,14 +655,6 @@ def _snapshot_to_resolved(snap: pb.Snapshot) -> "WorkerResolvedRepo": _BG_THREAD_ADMIT_WAIT_S = 0.5 -def _bg_yield_enabled() -> bool: - """pgw#677: default ON; env kill switch for red-verification and - emergencies only. OFF restores the pre-fix shape: mint seeds idle-gate - + hold the bare run gate (inline compiles included) and shape-warm - compiles run ungated against tenant forwards.""" - return os.environ.get("GEN_WORKER_BG_YIELD", "1").strip() != "0" - - def _cell_execution_lane_matches( ref: str, family: str, @@ -2884,11 +2876,6 @@ def _delegated_pendings(pendings: typing.Mapping[int, Any]) -> bool: return any(getattr(p, "delegated", False) for p in pendings.values()) -def _eager_first_boot_enabled() -> bool: - """pgw#671: default ON; env kill switch for emergencies only.""" - return os.environ.get("GEN_WORKER_EAGER_FIRST_BOOT", "1").strip() != "0" - - @dataclass class _HostRamBlock: """One exact, still-unsatisfied host-RAM admission observation.""" @@ -9222,8 +9209,6 @@ def _eager_first_eligible( lane. A delegated pending's eager tier is the untouched pipeline itself; the router question belongs only to an in-process capture, whose eager-while-compiling routing is what a router performs.""" - if not _eager_first_boot_enabled(): - return False if not inj.pending_self_mints: return False if spec.cls is not None and callable(getattr(spec.cls, "warmup", None)): @@ -9706,22 +9691,6 @@ async def _unit(wj: Any) -> bool: in their own turns. False = preempted by a tenant arrival; the caller re-queues the unit.""" _checkpoint() - if not _bg_yield_enabled(): - # Legacy shape (kill switch / red-verification): idle-gate - # between units, bare run gate around the forward. - idle = asyncio.ensure_future(self.wait_idle()) - stop = asyncio.ensure_future(bg.abandon.wait()) - try: - await asyncio.wait( - {idle, stop}, return_when=asyncio.FIRST_COMPLETED) - finally: - for fut in (idle, stop): - if not fut.done(): - fut.cancel() - _checkpoint() - async with rec.run_lock: - await _forward(wj) - return True async with self._bg_turn(rec, "seed", abort=bg.abandon) as stole: _checkpoint() try: @@ -11534,10 +11503,11 @@ def abort_check() -> None: def _wire_turn_gate(self, rec: _ClassRecord, pipeline: Any) -> None: """Hand this pipeline's hot-swap router the background-turn gate so every shape-warm/heal compile serializes with — and yields to — - tenant work (pgw#677). Idempotent; no-op without a router.""" - if not _bg_yield_enabled(): - return + tenant work (pgw#677). Idempotent; no-op without a router. + pgw#995: unconditional. ``GEN_WORKER_BG_YIELD`` used to be able to skip + this, restoring the pre-pgw#677 shape where shape-warm compiles ran + ungated against tenant forwards. Nothing ever set it.""" router = hot_swap.router_of(pipeline) if router is not None: router.set_turn_gate(self._bg_turn_threaded(rec)) diff --git a/src/gen_worker/fleet_cells.py b/src/gen_worker/fleet_cells.py index 678d2eca..fe10c3b7 100644 --- a/src/gen_worker/fleet_cells.py +++ b/src/gen_worker/fleet_cells.py @@ -1851,9 +1851,10 @@ def _unregister(pending: "PendingSelfMint") -> None: #: off") that named two causes which were BOTH false on the measured pod while #: the true cause — the pipeline-side mandatory-lane misclassification — was #: not named at all. A refusal that cannot name its own cause is the defect. +#: pgw#995 dropped `eager_first_disabled`: eager-first is unconditional, so +#: that cause can no longer arise and a reason nobody can reach is dead prose. _DELEGATION_DECLINE_PHASE = { "mint_in_process_forced": "aot_mint_forced_in_process", - "eager_first_disabled": "aot_eager_first_disabled", "no_eager_tier": "aot_no_eager_tier", "caller_forced_in_process": "aot_mint_forced_in_process", } @@ -1862,9 +1863,6 @@ def _unregister(pending: "PendingSelfMint") -> None: "GEN_WORKER_MINT_IN_PROCESS is set, which forces the in-process " "capture; an AOTI export has no eager tier to serve from while it " "compiles, so it cannot ride that shape", - "eager_first_disabled": - "GEN_WORKER_EAGER_FIRST_BOOT=0 turned eager-first off, and delegation " - "IS eager-first — there is no route to serve while a child compiles", "no_eager_tier": "an armed non-eager backend (AOTI cell or TRT engine) has replaced " "this pipeline's forward, so there is no eager tier to serve from", diff --git a/src/gen_worker/mint_delegate.py b/src/gen_worker/mint_delegate.py index 5d1656a0..42a6a33b 100644 --- a/src/gen_worker/mint_delegate.py +++ b/src/gen_worker/mint_delegate.py @@ -48,32 +48,41 @@ #: Kill switch, for red-verifying the in-process shape only. Delegation is the #: default because the in-process shape VIOLATES the liveness contract #: (WORKER-CONTRACTS §1/§2) — it is kept reachable to prove that, not to run. +#: +#: pgw#995: this is the LAST surviving env behaviour switch in this repo whose +#: default is ON and whose fleet declaration count is zero — i.e. the last one +#: that is safe to delete — and it is not deleted here. The reason is specific, +#: not a shrug: ``fleet_cells.enable_compiled`` ALREADY takes ``delegate=False`` +#: and reports it as ``caller_forced_in_process``, so a strictly better +#: parameter seam exists and this env is redundant with it. What blocks the +#: deletion is that ten test sites across seven files force the shape by setting +#: this env and then driving the EXECUTOR, which reaches the policy with +#: ``delegate=None``; threading the parameter through those call sites is a real +#: refactor of a 4k-line test file and does not belong in a sweep. Owner and +#: scope are recorded on pgw#995. Do not add a second reader in the meantime. ENV_IN_PROCESS = "GEN_WORKER_MINT_IN_PROCESS" #: Typed refusals :func:`delegation_refusal` can return. They are the OPERATOR -#: half of the decision (env kill switches); the PIPELINE half lives in +#: half of the decision; the PIPELINE half lives in #: ``fleet_cells.delegation_refusal``. pgw#813: the two were collapsed into one #: either/or sentence on the wire, so a real refusal named two causes that were #: both false and never named the one that was true. +#: +#: pgw#995 deleted ``REFUSAL_EAGER_FIRST_DISABLED`` with its switch. Delegation +#: IS eager-first, so the two moved together — and once eager-first is +#: unconditional, "eager-first is off" is not a state this worker can be in. +#: A refusal reason that can never be returned is a cause a reader will hunt for +#: and never find, which is the same defect as a cause that goes unnamed. REFUSAL_IN_PROCESS_FORCED = "mint_in_process_forced" -REFUSAL_EAGER_FIRST_DISABLED = "eager_first_disabled" def delegation_refusal() -> str: - """"" when this WORKER may mint out of process, else the typed reason. - - Delegation IS eager-first: the live pipeline is never armed, so a boot that - has eager-first turned off has no route to serve while a child compiles. - The two switches therefore move together rather than combining into a - fourth, undefined shape. - """ + """"" when this WORKER may mint out of process, else the typed reason.""" if os.environ.get(ENV_IN_PROCESS, "").strip().lower() in ( "1", "true", "yes", "on" ): return REFUSAL_IN_PROCESS_FORCED - if os.environ.get("GEN_WORKER_EAGER_FIRST_BOOT", "1").strip() == "0": - return REFUSAL_EAGER_FIRST_DISABLED return "" diff --git a/tests/harness/hub_env.py b/tests/harness/hub_env.py new file mode 100644 index 00000000..5e28880f --- /dev/null +++ b/tests/harness/hub_env.py @@ -0,0 +1,155 @@ +"""pgw#995 — the hub's env-delivery chain, modelled so local tests can see it. + +THE BLIND SPOT THIS CLOSES. `scripts/micro_mint_rig.py` (pgw#978) runs the whole +mint machinery on this box and is why a change can be proven before PyPI. But it +**constructs its own environment**: the mint child gets `mint_process.child_env` +plus a few rig keys, and the adopting process gets `dict(os.environ)`. Neither +resembles how a production pod is given its env, so the chain below was +invisible to every test in this repo: + + worker function declares env -> build schema -> release_env_declarations + operator sets a value -> endpoint_env_entries (+ Vault) + pod launch -> EndpointEnvService.Resolve -> pod env + | + config.loader ----+--> Settings + +That chain is exactly what took `GEN_WORKER_PREFER_AOT` dark. The flag was +declared by the worker function and set on the endpoint; a release rebuild +stopped declaring the name, the hub withheld every matching entry **silently**, +and three pod attempts went by before anyone noticed the AOT path was off. Every +component was individually correct. The DELIVERY was what broke, and delivery +was the one thing nothing tested. + +WHAT THIS IS. A faithful model of `EndpointEnvService.Resolve`'s contract — NOT +a reimplementation of the hub. It carries the one rule that matters (an entry +reaches the pod only if the release DECLARES its name) and the reserved-name +defence, and it reports withholdings with the same typed vocabulary the hub uses +(th#1650). It is deliberately small: a big fake hub would drift from the real +one and start certifying its own behaviour. + +WHAT IT IS NOT. It does not model Vault, `applies_to` version/tag matching, or +the mTLS resolve path. Those belong to the full HelloAck-shaped boot filed on +pgw#995; this is the seam plus the first regression it makes visible. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Dict, Iterable, List, Mapping, Sequence, Tuple + +#: Mirrors tensorhub `internal/api/endpoint_env_withheld.go` (th#1650). Kept +#: verbatim so a log line from a real pod and a failure from this harness use +#: the same word for the same thing. +WITHHELD_UNDECLARED = "undeclared_by_release" +WITHHELD_RESERVED = "reserved_name" + +#: Mirrors the hub's `reservedEnvPrefixes` narrowly — only what a worker test +#: can meaningfully trip. The hub is the authority; this exists so a test that +#: sets a platform name gets the platform answer rather than a false pass. +RESERVED_PREFIXES: Tuple[str, ...] = ( + "WORKER_", "TENSORHUB_", "ORCHESTRATOR_", "HF_HOME", "RUNPOD_", + "GEN_WORKER_C2PA_", "GEN_WORKER_PROCESS_SPLIT", "GEN_WORKER_COMPUTE_CHILD", + "GEN_WORKER_CHILD_", +) + + +@dataclass(frozen=True) +class Withheld: + """One entry the operator set that this pod will not receive.""" + + name: str + reason: str + detail: str = "" + + +@dataclass +class ReleaseEnvDeclarations: + """What THIS release's worker functions declare, per `release_env_declarations`. + + Per-RELEASE, which is the property that makes the postmortem possible: a + rebuild produces a new declaration set while the operator's entries are + per-ENDPOINT and long-lived. + """ + + names: Tuple[str, ...] = () + + @classmethod + def of(cls, *names: str) -> "ReleaseEnvDeclarations": + return cls(tuple(names)) + + +@dataclass +class EndpointEnvEntries: + """What the operator set, per `endpoint_env_entries`. Long-lived.""" + + values: Dict[str, str] = field(default_factory=dict) + + +@dataclass(frozen=True) +class Delivery: + """The outcome of one pod launch's env resolution.""" + + env: Dict[str, str] + withheld: Tuple[Withheld, ...] + + def withheld_names(self) -> List[str]: + return [w.name for w in self.withheld] + + +def is_reserved(name: str) -> bool: + return any(name.startswith(p) for p in RESERVED_PREFIXES) + + +def resolve( + declarations: ReleaseEnvDeclarations, + entries: EndpointEnvEntries, +) -> Delivery: + """The hub's rule: an entry reaches the pod only if the release declares it. + + Returns the delivered map AND every withholding, because a delivery that + reports only what arrived is the exact shape that hid the defect — "nothing + arrived" and "nothing was ever configured" have to be distinguishable. + """ + declared = set(declarations.names) + env: Dict[str, str] = {} + withheld: List[Withheld] = [] + for name in sorted(entries.values): + if name not in declared: + withheld.append(Withheld( + name, WITHHELD_UNDECLARED, + f"release declares {len(declared)} env name(s), not this one")) + continue + if is_reserved(name): + withheld.append(Withheld( + name, WITHHELD_RESERVED, "platform-reserved namespace")) + continue + env[name] = entries.values[name] + return Delivery(env=env, withheld=tuple(withheld)) + + +def pod_environ( + base: Mapping[str, str], + delivery: Delivery, + *, + strip: Iterable[str] = (), +) -> Dict[str, str]: + """The environment a pod actually boots with: image env + delivered entries. + + `strip` removes names the launching process happens to carry — without it a + rig running on a developer box would let an ambient export stand in for a + hub-delivered value and prove nothing. That substitution IS the blind spot; + a rig mode that allowed it would be decoration. + """ + out = {k: v for k, v in base.items() if k not in set(strip)} + out.update(delivery.env) + return out + + +def declared_by(function_env: Sequence[str]) -> ReleaseEnvDeclarations: + """Build declarations the way a build does: from the function's own list. + + Real builds read this off the function schema payload. A worker function + that stops listing a name produces a release that does not declare it — + with no error anywhere, because an empty `env` list is legal. + """ + return ReleaseEnvDeclarations(tuple(function_env)) diff --git a/tests/test_aot_mint_unblock_pgw813_pgw815.py b/tests/test_aot_mint_unblock_pgw813_pgw815.py index d62316dd..e28a3841 100644 --- a/tests/test_aot_mint_unblock_pgw813_pgw815.py +++ b/tests/test_aot_mint_unblock_pgw813_pgw815.py @@ -4,7 +4,7 @@ aot_cell_discovery miss family=sdxl lane=w8a8-lora64 self_mint_skipped aot_requires_delegation "out-of-process minting is - disabled (GEN_WORKER_MINT_IN_PROCESS or eager-first + disabled (GEN_WORKER_MINT_IN_PROCESS off) and an AOTI export has no eager tier..." self_mint_started dynamo ... armed an in-process capture @@ -272,19 +272,22 @@ def test_delegation_declines_name_their_TRUE_cause( _arm() assert "aot_mint_forced_in_process" in _phases(_events, "self_mint_skipped") + # pgw#995: the second arm here drove `GEN_WORKER_EAGER_FIRST_BOOT=0` and + # asserted the `aot_eager_first_disabled` phase. Both the switch and the + # phase are deleted — eager-first is unconditional, so that decline cannot + # arise, and a reason nobody can reach is a cause a reader hunts for and + # never finds. The pgw#813 claim under test is unharmed: it is that a + # refusal names its TRUE cause, which the operator arm above and the + # pipeline arm below still exercise. _events.clear() monkeypatch.delenv("GEN_WORKER_MINT_IN_PROCESS") - monkeypatch.setenv("GEN_WORKER_EAGER_FIRST_BOOT", "0") fleet_cells._PENDING.clear() - _arm() - assert "aot_eager_first_disabled" in _phases(_events, "self_mint_skipped") # pgw#846: `Compile.regional` is the dynamo/JIT per-block knob (ie#381) # and the AOT mint ignores it — regional EXPORT is retired, the recipe is # always whole-graph. A family that declares it must neither decline # delegation nor change the mint shape. _events.clear() - monkeypatch.delenv("GEN_WORKER_EAGER_FIRST_BOOT") fleet_cells._PENDING.clear() fleet_cells.enable_compiled( _Pipe(), _Cfg(regional=True), publisher=_Publisher()) # type: ignore[arg-type] @@ -296,11 +299,14 @@ def test_mint_delegate_names_its_own_refusals( monkeypatch: pytest.MonkeyPatch, ) -> None: monkeypatch.delenv("GEN_WORKER_MINT_IN_PROCESS", raising=False) - monkeypatch.delenv("GEN_WORKER_EAGER_FIRST_BOOT", raising=False) assert mint_delegate.delegation_refusal() == "" + # pgw#995: eager-first is unconditional, so setting the deleted name is a + # no-op rather than a second refusal. Asserted, not assumed — a deletion + # that leaves a live reader somewhere else looks exactly like this test + # passing for the wrong reason. monkeypatch.setenv("GEN_WORKER_EAGER_FIRST_BOOT", "0") - assert (mint_delegate.delegation_refusal() - == mint_delegate.REFUSAL_EAGER_FIRST_DISABLED) + assert mint_delegate.delegation_refusal() == "" + assert not hasattr(mint_delegate, "REFUSAL_EAGER_FIRST_DISABLED") monkeypatch.setenv("GEN_WORKER_MINT_IN_PROCESS", "1") assert (mint_delegate.delegation_refusal() == mint_delegate.REFUSAL_IN_PROCESS_FORCED) @@ -342,7 +348,6 @@ def test_eager_first_admits_a_DELEGATED_pending_with_no_router( lines later — pgw#784 could not run on any lane.""" from gen_worker.models import loading as loading_mod - monkeypatch.delenv("GEN_WORKER_EAGER_FIRST_BOOT", raising=False) monkeypatch.setattr( loading_mod, "pipeline_weight_lane", lambda p: "w8a8-lora64") @@ -371,7 +376,6 @@ def test_eager_first_still_requires_a_router_for_an_IN_PROCESS_capture( router, so no router means no eager tier.""" from gen_worker.models import loading as loading_mod - monkeypatch.delenv("GEN_WORKER_EAGER_FIRST_BOOT", raising=False) monkeypatch.setattr(loading_mod, "pipeline_weight_lane", lambda p: "") ex = _executor(tmp_path) diff --git a/tests/test_aot_wrapper_split_pgw793.py b/tests/test_aot_wrapper_split_pgw793.py index e090ca1f..2249fc95 100644 --- a/tests/test_aot_wrapper_split_pgw793.py +++ b/tests/test_aot_wrapper_split_pgw793.py @@ -420,11 +420,20 @@ def digests() -> tuple[str, str]: ws._installed = False -def test_install_is_idempotent_and_killable(monkeypatch) -> None: # type: ignore[no-untyped-def] +def test_install_is_idempotent_and_no_env_can_kill_it(monkeypatch) -> None: # type: ignore[no-untyped-def] + """pgw#995: the `GEN_WORKER_AOT_WRAPPER_SPLIT_OFF` arm is DELETED. + + It defaulted ON, no release ever declared it and no endpoint ever set it, so + deleting it made the shape every pod already ran unconditional. Setting the + retired name must now be inert — asserted rather than assumed, because a + deletion that leaves a live reader elsewhere looks exactly like this test + passing for the wrong reason. + """ monkeypatch.setattr(ws, "_installed", False) - monkeypatch.setenv(ws.DISABLE_ENV, "1") - assert ws.install() is False - monkeypatch.delenv(ws.DISABLE_ENV) + monkeypatch.setenv("GEN_WORKER_AOT_WRAPPER_SPLIT_OFF", "1") + assert not hasattr(ws, "DISABLE_ENV"), ( + "the v1 kill switch is back; env carries config and secrets, never a " + "branch selection") torch_inductor = pytest.importorskip("torch._inductor.cpp_builder") original = torch_inductor.run_compile_cmd try: diff --git a/tests/test_eager_first_boot_pgw671.py b/tests/test_eager_first_boot_pgw671.py index 5bd6a1b7..1eb9d50b 100644 --- a/tests/test_eager_first_boot_pgw671.py +++ b/tests/test_eager_first_boot_pgw671.py @@ -311,40 +311,79 @@ async def _run() -> None: asyncio.run(_run()) -def test_kill_switch_restores_the_sequential_gate_and_measures_the_split( +def test_eager_first_is_unconditional_and_no_env_restores_the_sequential_gate( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - """RED-VERIFICATION of the gate removal: with the env kill switch the - old ladder is back — every compile lands BEFORE READY — and the - elapsed-boot split is the eager-vs-compiled latency evidence.""" + """pgw#995 RED-VERIFICATION, inverted: the surviving path is the ONLY path. + + This test used to drive `GEN_WORKER_EAGER_FIRST_BOOT=0` to restore the + pre-pgw#671 sequential ladder, and the elapsed-boot split between the two + arms was the eager-vs-compiled latency evidence. That switch is deleted: + it defaulted ON, no release ever declared it and no endpoint ever set it, + so deleting it made the shape every pod already ran unconditional. + + An env used as a red-verification seam is the same defect as an env used as + a feature gate — it is a behaviour switch that a rebuild can flip, and + `GEN_WORKER_PREFER_AOT` is what that costs. So the assertion inverts: rather + than proving the OFF arm still works, prove the OFF arm is UNREACHABLE. + Setting the old name (or any plausible spelling of it) must change nothing. + + The latency evidence the old second arm carried is not lost — it is the + absolute bound below, which is the half that actually stated the claim: + time-to-READY does not pay the compile wall. The old test asserted BOTH + `ready_at < 3 * delay` and `ready_at < h_off.ready_at`; the second is + implied by the first whenever the sequential arm is honest, so the arm was + paying for a comparison the bound already made. + """ delay = 0.3 - monkeypatch.setenv("GEN_WORKER_EAGER_FIRST_BOOT", "0") - h_off = _Harness(tmp_path / "off", monkeypatch, compile_delay_s=delay) - - async def _off() -> None: - await h_off.boot() - assert h_off.rec.ready is True - assert h_off.rec.background_mint is None - # Sequential: the full plan compiled foreground, gating READY. - assert len(h_off.compile_log) == 3 - assert h_off.ex.serving_tiers() == {"generate": "compiled"} - - asyncio.run(_off()) - assert h_off.ready_at is not None and h_off.ready_at >= 3 * delay - - monkeypatch.setenv("GEN_WORKER_EAGER_FIRST_BOOT", "1") - h_on = _Harness(tmp_path / "on", monkeypatch, compile_delay_s=delay) - - async def _on() -> None: - await h_on.boot() - assert h_on.rec.background_mint is not None - await h_on.wait_mint() - - asyncio.run(_on()) - assert h_on.ready_at is not None - # The whole point: time-to-READY no longer pays the compile wall. - assert h_on.ready_at < 3 * delay - assert h_on.ready_at < h_off.ready_at + for name in ( + "GEN_WORKER_EAGER_FIRST_BOOT", + "GEN_WORKER_EAGER_FIRST", + "GEN_WORKER_EAGER_FIRST_BOOT_OFF", + ): + monkeypatch.setenv(name, "0") + + h = _Harness(tmp_path / "on", monkeypatch, compile_delay_s=delay) + + async def _run() -> None: + await h.boot() + # The pre-pgw#671 ladder would have compiled the full plan in the + # foreground and left no background mint at all. + assert h.rec.background_mint is not None, ( + "an env turned eager-first off — the pgw#995 deletion did not take, " + "or a new switch was added on top of it") + assert h.rec.ready is True + await h.wait_mint() + + asyncio.run(_run()) + assert h.ready_at is not None + # The whole point, stated as an absolute bound rather than a comparison + # against an arm that no longer exists: READY does not wait for the wall. + assert h.ready_at < 3 * delay + + +def test_no_env_read_survives_in_the_eager_first_and_bg_yield_paths() -> None: + """pgw#995 structural guard: the deleted names are gone from the SOURCE. + + A behavioural assertion can pass while a second, unreached reader survives + somewhere else in the module — which is exactly how `GEN_WORKER_PREFER_AOT` + kept two live gates after one was believed removed. So read the source. + """ + import gen_worker.executor as _ex + import gen_worker.mint_delegate as _md + import gen_worker.aot_wrapper_split as _ws + + src = "".join( + Path(m.__file__).read_text() for m in (_ex, _md, _ws) if m.__file__) + for gone in ( + "GEN_WORKER_EAGER_FIRST_BOOT", + "GEN_WORKER_BG_YIELD", + "GEN_WORKER_AOT_WRAPPER_SPLIT_OFF", + ): + # Prose that NAMES the deleted switch is fine and wanted; a read is not. + assert f'environ.get("{gone}"' not in src, ( + f"{gone} is read again — pgw#995 deleted it because env must carry " + f"config and secrets, never a branch selection") def test_mid_build_abandonment_is_clean_and_keeps_serving_eager( diff --git a/tests/test_hub_env_delivery_pgw995.py b/tests/test_hub_env_delivery_pgw995.py new file mode 100644 index 00000000..a68501c8 --- /dev/null +++ b/tests/test_hub_env_delivery_pgw995.py @@ -0,0 +1,241 @@ +"""pgw#995 deliverable 2 — hub-shaped env delivery reaches `Settings`. + +The `GEN_WORKER_PREFER_AOT` postmortem was not a bug in any component. The flag +was declared, the entry was set, the loader worked, the gate worked. What broke +was the DELIVERY between them: a release rebuild stopped declaring the name, the +hub withheld the entry silently, and the worker booted without it. Three pod +attempts. + +Nothing in this repo could have caught that, because the local rig constructs +its own environment — `mint_process.child_env` for the mint child, +`dict(os.environ)` for the adopting process. Both are shapes a production pod +never has. So the regression class had exactly one detector: a pod. + +These tests give it a second one. They drive the REAL +`gen_worker.config.load_settings` — the same function `entrypoint._run_main` +calls — over an environment produced by the hub's resolution rule rather than by +the test's own convenience. + +Run: pytest tests/test_hub_env_delivery_pgw995.py -v +""" + +from __future__ import annotations + +import pytest + +from gen_worker import config as config_pkg +from gen_worker.config import load_settings + +from harness import hub_env + + +def _boot(monkeypatch: pytest.MonkeyPatch, env: dict) -> None: + """Replace the process environment with a pod's, exactly.""" + for name in list(os_environ_names()): + monkeypatch.delenv(name, raising=False) + for k, v in env.items(): + monkeypatch.setenv(k, v) + + +def os_environ_names() -> list: + import os + + return [n for n in os.environ if n.startswith( + ("GEN_WORKER_", "TENSORHUB_", "WORKER_", "COZY_", "HF_"))] + + +# --------------------------------------------------------------------------- +# The seam: a DECLARED entry is delivered and reaches the typed struct +# --------------------------------------------------------------------------- + + +def test_a_hub_delivered_env_value_reaches_settings( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The whole chain, end to end, with nothing hand-placed in the middle. + + The worker function declares `HF_TOKEN`; the operator sets it; the release + declares it; so the hub delivers it and `Settings` carries it. `HF_TOKEN` is + the realistic subject rather than a made-up name: it is org-settable (the + hub reserves the `HF_` prefix but exempts this one), it is what a gated + clone needs, and its absence is what th#1073 records breaking gated clones + for days. + + Every hop is the production one — `resolve()` implements the hub's rule and + `load_settings()` is the function the worker's own entrypoint calls. + """ + declarations = hub_env.declared_by(["HF_TOKEN"]) + entries = hub_env.EndpointEnvEntries( + {"HF_TOKEN": "hf_operator_set_token"}) + + delivery = hub_env.resolve(declarations, entries) + assert delivery.env == {"HF_TOKEN": "hf_operator_set_token"} + assert delivery.withheld == () + + # The pod boots with image env + delivered entries, and NOTHING the test + # process happened to be carrying. + _boot(monkeypatch, hub_env.pod_environ({}, delivery)) + + settings = load_settings() + assert settings.hf_token == "hf_operator_set_token", ( + "a declared, delivered env did not reach Settings — the delivery chain " + "is broken between the hub's resolve and config.loader") + + +# --------------------------------------------------------------------------- +# The regression: the postmortem's exact shape, now RED locally +# --------------------------------------------------------------------------- + + +def test_a_rebuild_that_stops_declaring_a_name_withholds_it_and_says_so( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """`GEN_WORKER_PREFER_AOT`, reproduced in milliseconds instead of three pods. + + Release N declares the name and the value arrives. Release N+1 is rebuilt + from worker code that no longer declares it — the operator changed nothing, + the entry is untouched — and the value stops arriving. The point is not that + it stops (that is the declaration contract working); the point is that the + delivery SAYS SO, which is the half that was missing on both sides. + """ + entries = hub_env.EndpointEnvEntries( + {"HF_TOKEN": "hf_operator_set_token"}) + + before = hub_env.resolve( + hub_env.declared_by(["HF_TOKEN"]), entries) + _boot(monkeypatch, hub_env.pod_environ({}, before)) + assert load_settings().hf_token == "hf_operator_set_token" + + # The rebuild. Nobody edited the entry; the worker function's env list + # changed, so the release declares nothing. + after = hub_env.resolve(hub_env.declared_by([]), entries) + + assert after.env == {}, "an undeclared name must not be injected" + assert after.withheld_names() == ["HF_TOKEN"], ( + "the rebuild dropped a configured entry and reported NOTHING — this is " + "precisely the silence that cost three pod attempts (th#1650)") + assert after.withheld[0].reason == hub_env.WITHHELD_UNDECLARED + assert "0 env name(s)" in after.withheld[0].detail, ( + "a withholding must distinguish 'this release declares nothing at all' " + "(the rebuild case) from 'this one name was removed' (the intended one)") + + _boot(monkeypatch, hub_env.pod_environ({}, after)) + assert load_settings().hf_token == "", ( + "the withheld value still reached Settings — something other than the " + "hub is supplying it, which is the substitution this harness exists to " + "forbid") + + +def test_an_ambient_export_cannot_stand_in_for_a_hub_delivered_value( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A rig that lets the developer's own shell satisfy the assertion is + decoration. `pod_environ(strip=...)` is what stops that, so it is tested + rather than trusted: with the name exported ambiently and NOT declared by + the release, `Settings` must still come up empty. + """ + monkeypatch.setenv("HF_TOKEN", "ambient_shell_token") + ambient = {"HF_TOKEN": "ambient_shell_token"} + + delivery = hub_env.resolve( + hub_env.declared_by([]), + hub_env.EndpointEnvEntries({"HF_TOKEN": "hf_operator_set_token"})) + + env = hub_env.pod_environ(ambient, delivery, strip=["HF_TOKEN"]) + _boot(monkeypatch, env) + assert load_settings().hf_token == "" + + +# --------------------------------------------------------------------------- +# The reserved namespace still wins over a declaration +# --------------------------------------------------------------------------- + + +def test_a_release_cannot_declare_its_way_into_the_platform_namespace( +) -> None: + """pgw#763 delta 0: the process-split switch is platform-only. Declaring it + must not deliver it — otherwise a release could opt its own tenant code out + of the boundary that contains it.""" + delivery = hub_env.resolve( + hub_env.declared_by(["GEN_WORKER_COMPUTE_CHILD"]), + hub_env.EndpointEnvEntries({"GEN_WORKER_COMPUTE_CHILD": "1"})) + assert delivery.env == {} + assert delivery.withheld[0].reason == hub_env.WITHHELD_RESERVED + + +def test_the_loader_is_the_only_component_this_harness_talks_to() -> None: + """Guard against the harness growing into a second config implementation. + + `hub_env` models DELIVERY. The moment it starts constructing `Settings` + itself it stops testing the production path and starts certifying its own — + which is how a harness becomes the thing it was written to check. + """ + src = (hub_env.__file__ or "") + assert src + text = open(src).read() + for forbidden in ("load_settings", "Settings(", "msgspec"): + assert forbidden not in text, ( + f"hub_env references {forbidden!r}: it must produce an ENVIRONMENT " + f"and let the real loader turn it into config") + assert config_pkg.load_settings is load_settings + + +# --------------------------------------------------------------------------- +# The rig mode itself (pgw#995 deliverable 2) +# --------------------------------------------------------------------------- + + +def _rig(): + import sys + from pathlib import Path + + root = Path(__file__).resolve().parents[1] + sys.path.insert(0, str(root / "scripts")) + import micro_mint_rig # noqa: PLC0415 - the script under test + + return micro_mint_rig + + +def test_rig_hub_env_mode_delivers_declared_and_strips_ambient() -> None: + """`--hub-env` boots the mint child the way a pod is booted. + + Two properties, and the second is the one that makes the mode worth having: + a DECLARED entry is delivered, and an AMBIENT value of the same name is + stripped rather than inherited. Without the strip, a developer with the + variable exported in their shell gets a green rig for a release that would + have booted without it on a pod — which is not a weaker test, it is a test + that reports the opposite of the truth. + """ + rig = _rig() + env, withheld = rig.hub_delivered_env( + {"PATH": "/usr/bin", "HF_TOKEN": "ambient-shell-value"}, + {"HF_TOKEN": "delivered-by-hub"}) + + assert env["HF_TOKEN"] == "delivered-by-hub" + assert env["PATH"] == "/usr/bin", "image env must survive" + assert withheld == [] + + +def test_rig_hub_env_mode_reports_an_undeclared_entry_instead_of_dropping_it( +) -> None: + """The rig's whole reason to exist is turning a pod-only failure into a + local one. An entry the release does not declare must show up as a FACT the + rig reports, not as a variable that quietly is not there.""" + rig = _rig() + env, withheld = rig.hub_delivered_env( + {"PATH": "/usr/bin"}, {"COZY_SOMETHING_UNDECLARED": "x"}) + + assert "COZY_SOMETHING_UNDECLARED" not in env + assert len(withheld) == 1 + assert withheld[0]["name"] == "COZY_SOMETHING_UNDECLARED" + assert withheld[0]["reason"] == hub_env.WITHHELD_UNDECLARED + assert withheld[0]["detail"], "a withholding with no detail is a shrug" + + +def test_rig_strips_every_name_it_claims_to_deliver() -> None: + """A name the rig DECLARES but does not STRIP is a hole: ambient value in, + hub value never exercised, mode silently decorative.""" + rig = _rig() + assert set(rig.RIG_DECLARED_ENV) <= set(rig.RIG_STRIPPED_ENV), ( + "every declared name must also be stripped, or the ambient environment " + "can satisfy the assertion the mode exists to make") diff --git a/tests/test_mint_gate_pgw677.py b/tests/test_mint_gate_pgw677.py index 2509ff57..bfa9c92c 100644 --- a/tests/test_mint_gate_pgw677.py +++ b/tests/test_mint_gate_pgw677.py @@ -4,19 +4,23 @@ mint work yields: 1. STARVATION SHAPE (the live incident): during a background mint, - tenant requests complete at serving latency. RED-verified via the - GEN_WORKER_BG_YIELD=0 kill switch, which restores the pre-fix tree: - mint seed units inline-compile while holding the per-instance run - gate (the router's headroom degrade), so a tenant request queues for - the length of the unit — the measured "completions land exactly as - mint units free the gate / 0 renders in 19 min" shape. + tenant requests complete at serving latency. This used to be + RED-verified against the pre-fix tree, reachable via a + GEN_WORKER_BG_YIELD=0 kill switch: mint seed units inline-compiled + while holding the per-instance run gate (the router's headroom + degrade), so a tenant request queued for the length of the unit — + the measured "completions land exactly as mint units free the gate / + 0 renders in 19 min" shape. pgw#995 DELETED that switch and the tree + it selected, so the shape is now unreachable rather than merely + unselected, and the property is asserted in absolute terms against + the harness's own configured quantities. 2. RACE EXCLUSION (the pgw#676 SIGSEGV class): the shape-warm thread's compile can never execute the shared modules concurrently with a tenant forward. Post-fix the compile owns a background turn (single-flight, instance turn_mutex, tenant-quiet admission); the tenant that arrives mid-compile waits — bounded by ONE compile — and its wait is attributed to `instance_gate_wait`, never to runtime_ms. - RED-verified: with the kill switch the overlap is observed. + Structurally impossible post-pgw#995, not merely not-selected. 3. MINIMUM PROGRESS: under a sustained tenant stream the mint still finishes — the steal rule grants one bounded background unit per debt window; stolen units are not preemptible. @@ -328,38 +332,19 @@ def test_tenant_serves_at_serving_latency_during_mint_and_red_verifies( # headroom inside its exclusive turn instead. monkeypatch.setattr(hot_swap, "_headroom_ok", lambda device: False) - # --- RED: pre-fix tree via the kill switch --------------------------- - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "0") - h_off = _Harness( - tmp_path / "off", monkeypatch, - compile_delay_s=0.15, seed_forward_s=0.6, tenant_forward_s=0.02) + # pgw#995: the RED arm here drove `GEN_WORKER_BG_YIELD=0` to restore the + # pre-pgw#677 tree (idle-gate between units, bare run gate around the + # forward, inline compiles on handler workers) and asserted the starvation + # shape it produced. The switch is deleted — it defaulted ON, no release + # ever declared it and no endpoint ever set it — so the legacy tree it + # selected is deleted with it and there is nothing left to red-verify + # against. An env kept alive to be a test seam is still an env that selects + # behaviour, and a release rebuild cannot tell the two purposes apart. + # + # What survives is the arm that states the property in absolute terms, and + # `test_no_env_restores_the_pre_pgw677_tree` below proves the deleted arm is + # unreachable rather than merely unused. - async def _off() -> Tuple[float, pb.JobResult]: - await h_off.boot() - assert h_off.rec.background_mint is not None - # Wait until a BACKGROUND mint seed unit is in flight, holding the - # run gate (the boot's foreground eager pass contributed run #1). - deadline = time.monotonic() + 10.0 - while len(h_off.seed_runs) < 2 and time.monotonic() < deadline: - await asyncio.sleep(0.01) - assert len(h_off.seed_runs) >= 2, "mint never started seeding" - res, wall = await h_off.dispatch("r-red", aspect="16:9") - assert res.status == pb.JOB_STATUS_OK, res.safe_message - return wall, res - - wall_off, _res_off = asyncio.run(_off()) - # The collapse shape: the request queued behind the seed unit's - # inline compile (and then paid its own inline compile on the - # degraded router) — an order of magnitude over serving latency. - assert wall_off >= 0.5, ( - f"expected the pre-fix starvation shape, got {wall_off:.3f}s") - # Pre-fix, inline compiles ran on handler (to_thread) workers. - assert any( - "shape-warm" not in thread for _, thread, _, _ in h_off.compiles), ( - "pre-fix tree should compile inline off the warm thread") - - # --- POST-FIX --------------------------------------------------------- - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h_on = _Harness( tmp_path / "on", monkeypatch, compile_delay_s=0.15, seed_forward_s=0.6, tenant_forward_s=0.02) @@ -423,30 +408,14 @@ def test_compile_and_tenant_forward_never_overlap_and_red_verifies( segfaulted sm_86 (pgw#676: _forward_with_branch racing compile_wrapper) is structurally impossible — and the wait is attributed to the instance_gate_wait stage, not billed as runtime. RED: with the kill - switch the overlap is observed.""" - - # --- RED: pre-fix tree — overlap happens ----------------------------- - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "0") - h_off = _Harness( - tmp_path / "off", monkeypatch, - compile_delay_s=0.8, seed_forward_s=0.0, tenant_forward_s=0.05) - - async def _off() -> None: - await h_off.boot() - deadline = time.monotonic() + 10.0 - while not h_off.in_compile.is_set(): - assert time.monotonic() < deadline, "no background compile ran" - await asyncio.sleep(0.005) - res, _wall = await h_off.dispatch("r-race", aspect="1:1") - assert res.status == pb.JOB_STATUS_OK, res.safe_message + switch the overlap is observed (pgw#995: that arm is gone — see below).""" - asyncio.run(_off()) - assert h_off.overlaps, ( - "pre-fix tree must exhibit the pgw#676 overlap (tenant forward " - "concurrent with the warm thread's compile)") + # pgw#995: the RED arm drove `GEN_WORKER_BG_YIELD=0` and asserted that the + # pre-fix tree DOES exhibit the pgw#676 overlap. The switch and that tree + # are deleted, so the overlap is now structurally impossible rather than + # merely not-selected — which is the stronger statement the docstring + # already made. - # --- POST-FIX: exclusion holds, wait is attributed ------------------- - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h_on = _Harness( tmp_path / "on", monkeypatch, compile_delay_s=0.8, seed_forward_s=0.0, tenant_forward_s=0.05) @@ -492,7 +461,6 @@ def test_mint_completes_under_sustained_tenant_load( """A tenant stream that never drains would starve a purely idle-gated mint forever. The steal rule grants one bounded background unit per debt window: the mint completes while the stream keeps serving.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") monkeypatch.setattr(executor_mod, "_BG_STEAL_FLOOR_S", 0.1) monkeypatch.setattr(executor_mod, "_BG_STEAL_DEBT_FACTOR", 1.0) monkeypatch.setattr(executor_mod, "_BG_COMPILE_QUIESCENCE_S", 0.0) @@ -572,3 +540,25 @@ def compiled(*args: Any, **kwargs: Any) -> None: legacy.enable() verdict, _sig = legacy.route("t", compiled, ("d",), {}) assert verdict == hot_swap.COMPILED + + +# --------------------------------------------------------------------------- +# pgw#995 — the deleted arm is UNREACHABLE, not merely unused +# --------------------------------------------------------------------------- + + +def test_no_env_restores_the_pre_pgw677_tree() -> None: + """The two RED arms above were driven by an env. Deleting a switch and + deleting its tests looks identical to deleting a switch and leaving a + second reader behind — which is how `GEN_WORKER_PREFER_AOT` kept a live + gate after one was believed removed. So assert on the SOURCE. + """ + from pathlib import Path as _P + import gen_worker.executor as _ex + + src = _P(_ex.__file__).read_text() + assert 'environ.get("GEN_WORKER_BG_YIELD"' not in src, ( + "GEN_WORKER_BG_YIELD is read again — env carries config and secrets, " + "never a branch selection") + assert "_bg_yield_enabled" not in src, ( + "the bg-yield predicate is back; pgw#677's shape is unconditional") diff --git a/tests/test_mint_reopen_pgw677.py b/tests/test_mint_reopen_pgw677.py index f7bab584..5a60d3d3 100644 --- a/tests/test_mint_reopen_pgw677.py +++ b/tests/test_mint_reopen_pgw677.py @@ -333,7 +333,6 @@ def test_w8a8_stamp_with_hub_execution_lane_boots_eager_first( setup ran the FOREGROUND compile-then-serve mint (rec.background_mint is None), and the first tenant request sat behind the whole inline plan — the measured 26-minute cold-L4 starvation.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, compile_delay_s=0.4, seed_forward_s=0.05, tenant_forward_s=0.02, @@ -365,7 +364,6 @@ def test_true_mandatory_execution_lane_still_refuses_eager_first( """The qwen shape: hub lane says real w8a8 activations — eager is not a production tier there; the boot keeps the sequential foreground proof. Guards the fix from over-rotating.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, compile_delay_s=0.0, seed_forward_s=0.0, tenant_forward_s=0.01, @@ -384,7 +382,6 @@ def test_w8a8_stamp_without_execution_lane_evidence_stays_foreground( ) -> None: """No hub lane evidence: the weight-lane stamp remains the fail-closed fallback — unchanged pre-reopen behavior.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, weight_lane="w8a8", hub_execution_lane="") @@ -412,7 +409,6 @@ def test_multi_minute_compile_never_steals_against_live_demand( tenant completes at serving latency. RED on 0.70.0: the single 30s floor (patched to 0.1s, as tape 3 always did) lets the compile steal almost immediately and a tenant waits out the whole unabortable unit.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") monkeypatch.setattr(executor_mod, "_BG_STEAL_FLOOR_S", 0.1) # raising=False: on the pre-fix tree this attribute does not exist and # the single floor governs — that IS the red run. @@ -459,7 +455,6 @@ def test_compile_steal_is_announced_on_the_wire( """When the compile floor DOES elapse under truly continuous demand, the steal happens (minimum progress) and announces itself as a typed ``bg_turn_steal`` event.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") monkeypatch.setattr(executor_mod, "_BG_STEAL_FLOOR_S", 0.05) monkeypatch.setattr( executor_mod, "_BG_COMPILE_STEAL_FLOOR_S", 0.3, raising=False) @@ -503,7 +498,6 @@ def test_pack_or_closure_refusal_reaches_the_wire( (closure gate / pack) died in pod logs. Post-fix the verbatim reason rides the wire as a typed ``self_mint_abort`` event. RED on 0.70.0: no such event exists anywhere.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") def _refuse(*args: Any, **kwargs: Any) -> None: raise ValueError("closure refused: guard leak L['scale']") @@ -599,7 +593,6 @@ def test_oom_truncated_plan_never_finalizes_partial_capture( partial capture, and serving stays eager and alive. RED on 0.70.0: the OOM'd pass satisfied the stats-stable convergence and the mint finalized (phase=finalize) with nothing publishable and no event.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, compile_delay_s=0.02, seed_forward_s=0.01, tenant_forward_s=0.01, diff --git a/tests/test_mint_vram_budget_pgw737.py b/tests/test_mint_vram_budget_pgw737.py index 6fd972c3..6d3b9177 100644 --- a/tests/test_mint_vram_budget_pgw737.py +++ b/tests/test_mint_vram_budget_pgw737.py @@ -365,7 +365,6 @@ def test_mint_declines_on_a_card_that_cannot_hold_the_capture( tenant with it. Post-fix: no seed runs at all, one structured ``self_mint_skipped`` line on the wire, the tier stays eager, the cell stays absent, and the request SUCCEEDS.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness(tmp_path, monkeypatch, compile_delay_s=0.05) _card(monkeypatch, total_gib=79.19, resident_gib=54.2, peak_gib=65.4) @@ -404,7 +403,6 @@ def test_small_resident_still_mints_on_the_same_rig( ) -> None: """The fence: an sdxl-class residency has the headroom, so the budget must stay out of the way — the mint runs and the tier flips compiled.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness(tmp_path, monkeypatch, compile_delay_s=0.02) _card(monkeypatch, total_gib=23.6, resident_gib=7.0, peak_gib=9.5) @@ -432,7 +430,6 @@ def test_tenant_oom_evicts_the_mint_and_the_request_completes( five times and bought a second H100 for a deterministic failure. Post fix the MINT loses: it is abandoned, its targets unwrapped, the card freed, and the same request re-runs eager to OK on this same worker.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, compile_delay_s=1.5, seed_forward_s=0.05, tenant_oom_once=True)