From 88a445ef90d5be2ca837bad5fdcbb5f6c41905fa Mon Sep 17 00:00:00 2001 From: Paul Fidika Date: Thu, 6 Aug 2026 23:40:34 -0600 Subject: [PATCH 1/2] =?UTF-8?q?pgw#995:=20env=20carries=20values,=20never?= =?UTF-8?q?=20a=20branch=20selection=20=E2=80=94=20wave=202,=20plus=20the?= =?UTF-8?q?=20BEHAVIOUR=20axis,=20plus=20a=20rig=20that=20can=20see=20hub?= =?UTF-8?q?=20env=20delivery?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pgw#990 deleted GEN_WORKER_PREFER_AOT, the flag that gated both the mint recipe and cell discovery, silently disarmed on a release rebuild, and cost three pod attempts. This is the class rather than the instance. WHY THE EXISTING CENSUS COULD NOT HAVE CAUGHT IT. pgw#929/931 is a real census — 79 classified (file, variable) pairs — but all six of its classifications answer WHERE a read happens relative to the config pipeline, never WHETHER it selects behaviour. Those are orthogonal, and the postmortem lives entirely on the second axis. The proof it was missing rather than implicit is in the allowlist itself: GEN_WORKER_AOT_EXPORT_PARALLEL and _REUSE were filed as LIBRARY (torch has never heard of either name — they are first-party feature gates), and BG_YIELD / EAGER_FIRST_BOOT / MINT_IN_PROCESS as STANDALONE, "a CLI that loads no app config", while all three were read from executor.py on the serving hot path of every production pod. DELETED — three switches, each default-ON with zero fleet declarations, so deleting the env makes the shape every pod already ran unconditional and NO pod's behaviour changes: GEN_WORKER_BG_YIELD + the legacy idle-gate tree it selected GEN_WORKER_EAGER_FIRST_BOOT + REFUSAL_EAGER_FIRST_DISABLED, which can no longer be returned (a refusal reason nobody can reach is a cause a reader hunts for and never finds) GEN_WORKER_AOT_WRAPPER_SPLIT_OFF KEPT — and "delete every kill switch" is wrong in two separate directions: AOT_RUN_IMPL_SPLIT_OFF LIVE. 5 SDXL releases declare it, 1 endpoint carries a non-deleted entry. It sits in the same file as its deleted sibling and looks identical — zero declarations is a fact you MEASURE. AOT_EXPORT_PARALLEL default-OFF and dark by their own docstrings. Deleting AOT_EXPORT_REUSE those gates makes an unproven path unconditional, which is a LAUNCH, not a deletion. HOST_MOVE_GUARD, PROBE ruled exceptions with named threats. TESTS STOP USING ENV AS A RED-VERIFICATION SEAM. BG_YIELD=0 and EAGER_FIRST_BOOT=0 drove RED arms in four files. An env kept alive to be a test seam is still an env a release rebuild can flip. Those arms are replaced by absolute-bound assertions plus source-level guards proving the deleted arm is UNREACHABLE, not merely unused — deleting a switch and deleting its tests looks identical to deleting a switch and leaving a second reader behind, which is exactly how PREFER_AOT kept a live gate after one was believed removed. THE GUARD. lint_config_reads.py gains an AST pass that flags any env read feeding a conditional and fails the build unless the (file, variable) pair is in BEHAVIOUR_GATES *with the threat it defends against* — "it is useful" and "it is off by default" are rejected, and stale exemptions fail too. Same structural bar as the mint_recipe guard shipped in 0.93.2. RED-verified: adding a plausible new switch to executor.py fails with a message naming the postmortem. NOT DELETED, and it is the one remaining defect: GEN_WORKER_MINT_IN_PROCESS. Default-ON, zero declarations, and enable_compiled(delegate=False) is a strictly better parameter seam that already exists — blocked only on ten test sites across seven files that force the shape via the env and drive the executor. Recorded as a burn-down, not an exception. DELIVERABLE 2 — the rig blind spot. micro_mint_rig.py runs the whole mint machinery locally, which is why a change can be proven before PyPI, but it CONSTRUCTS ITS OWN ENVIRONMENT (mint_process.child_env for the mint child, dict(os.environ) for the adopting process) — shapes no production pod ever has. So the chain that actually delivers env to a pod (function declares -> release_env_declarations -> endpoint_env_entries -> Resolve -> pod env -> Settings) was invisible to every test in this repo, and the regression class that took PREFER_AOT dark had exactly one detector: a pod. tests/harness/hub_env.py models the hub's rule (an entry reaches the pod only if the release DECLARES its name, plus the reserved-namespace defence) and reports withholdings in th#1650's typed vocabulary. `--hub-env` boots the mint child through it and STRIPS ambient values, so a developer's shell cannot stand in for a hub-delivered one — that substitution is what the blind spot was made of, and a mode that allowed it would be decoration. tests/test_hub_env_delivery_pgw995.py drives the real config.load_settings() over hub-resolved environments and reproduces the postmortem in milliseconds: release N declares the name and the value reaches Settings; release N+1 is rebuilt without the declaration, nobody touches the entry, and the value is withheld AND says so. Full HelloAck-shaped boot (Vault, applies_to version/tag matching, mTLS resolve) is filed on pgw#995 with an owner. Hub half: th#1650. Gates: lint_config_reads OK (77 pairs), lint_http_timeouts OK, lint_unreached_surface OK, ruff clean on src/gen_worker and every changed file. --- changelog.d/pgw995.md | 71 ++++++ scripts/config_reads_allowlist.txt | 86 ++++--- scripts/lint_config_reads.py | 202 ++++++++++++++++ scripts/micro_mint_rig.py | 76 +++++- src/gen_worker/aot_wrapper_split.py | 23 +- src/gen_worker/config/loader.py | 3 - src/gen_worker/executor.py | 38 +-- src/gen_worker/fleet_cells.py | 6 +- src/gen_worker/mint_delegate.py | 31 ++- tests/harness/hub_env.py | 155 ++++++++++++ tests/test_aot_mint_unblock_pgw813_pgw815.py | 24 +- tests/test_eager_first_boot_pgw671.py | 101 +++++--- tests/test_hub_env_delivery_pgw995.py | 241 +++++++++++++++++++ tests/test_mint_gate_pgw677.py | 112 ++++----- tests/test_mint_reopen_pgw677.py | 7 - tests/test_mint_vram_budget_pgw737.py | 3 - 16 files changed, 972 insertions(+), 207 deletions(-) create mode 100644 changelog.d/pgw995.md create mode 100644 tests/harness/hub_env.py create mode 100644 tests/test_hub_env_delivery_pgw995.py diff --git a/changelog.d/pgw995.md b/changelog.d/pgw995.md new file mode 100644 index 00000000..acaf6bb1 --- /dev/null +++ b/changelog.d/pgw995.md @@ -0,0 +1,71 @@ +- **pgw#995: env carries values, never a branch selection — wave 2, plus the + BEHAVIOUR axis the first census could not see.** pgw#990 deleted + `GEN_WORKER_PREFER_AOT`, the flag that gated both the mint recipe and cell + discovery, silently disarmed on a release rebuild and cost three pod attempts. + This is the class rather than the instance. The pgw#929/931 census is real — + 79 classified (file, variable) pairs — but its six classifications all answer + **WHERE** a read happens relative to the config pipeline, never **WHETHER** it + selects behaviour, and the postmortem lives entirely on the second axis. The + evidence it was missing rather than implicit: `GEN_WORKER_AOT_EXPORT_PARALLEL` + and `_REUSE` were filed as **LIBRARY** (torch has never heard of either name — + they are first-party feature gates), and `GEN_WORKER_BG_YIELD`, + `_EAGER_FIRST_BOOT` and `_MINT_IN_PROCESS` as **STANDALONE** ("a CLI that loads + no app config") while all three were read from `executor.py` on the serving hot + path of every production pod. + **Three switches DELETED**, each default-ON with zero fleet declarations, so + deleting the env made the shape every pod already ran unconditional and no + pod's behaviour changed: `GEN_WORKER_BG_YIELD` (pgw#677 bg-yield, and the + legacy idle-gate tree it selected), `GEN_WORKER_EAGER_FIRST_BOOT` (pgw#671 + eager-first, and `mint_delegate.REFUSAL_EAGER_FIRST_DISABLED` with it — a + refusal reason that can no longer be returned is a cause a reader hunts for and + never finds), and `GEN_WORKER_AOT_WRAPPER_SPLIT_OFF`. + **Four gates KEPT, each with a stated reason, because "delete every kill + switch" is wrong in two different directions.** `GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF` + looks identical to its deleted sibling in the same file and is **LIVE** — 5 + SDXL releases declare it, 1 endpoint carries a non-deleted entry — so zero + declarations is a fact you MEASURE, not one you read off the code. + `GEN_WORKER_AOT_EXPORT_PARALLEL` / `_REUSE` are default-OFF and dark by their + own docstrings: deleting those gates would make an unproven path + unconditional, which is a **launch, not a deletion**. `GEN_WORKER_HOST_MOVE_GUARD` + and the pgw#980 probe pair are ruled exceptions with named threats. + **The tests stop using env as a red-verification seam.** `BG_YIELD=0` and + `EAGER_FIRST_BOOT=0` drove RED arms in four files; an env kept alive to be a + test seam is still an env a release rebuild can flip. Those arms are replaced + by absolute-bound assertions plus source-level guards proving the deleted arm + is **unreachable**, not merely unused — deleting a switch and deleting its + tests looks identical to deleting a switch and leaving a second reader behind, + which is exactly how PREFER_AOT kept a live gate after one was believed gone. + **`scripts/lint_config_reads.py` now enforces the behaviour axis.** An AST pass + flags any env read that feeds a conditional and fails the build unless the + (file, variable) pair is in `BEHAVIOUR_GATES` **with the threat it defends + against** — "it is useful" and "it is off by default" are rejected. Stale + exemptions fail too. Structurally the same bar as the `mint_recipe` guard + shipped in 0.93.2. + **`GEN_WORKER_MINT_IN_PROCESS` is NOT deleted, and it is the one remaining + defect.** Default-ON, zero declarations, and `enable_compiled(delegate=False)` + is a strictly better parameter seam that already exists — blocked only on ten + test sites across seven files that force the shape via the env and drive the + executor. Recorded in `BEHAVIOUR_GATES` as a burn-down, not an exception. + +- **pgw#995 deliverable 2: the local rig can finally see hub env delivery + (`micro_mint_rig.py --hub-env`).** The rig runs the whole mint machinery on + this box, which is why a change can be proven before PyPI — but it + **constructed its own environment** (`mint_process.child_env` for the mint + child, `dict(os.environ)` for the adopting process), shapes no production pod + ever has. So the chain that actually delivers env to a pod — worker function + declares → `release_env_declarations` → `endpoint_env_entries` → + `EndpointEnvService.Resolve` → pod env → `Settings` — was invisible to every + test in this repo, and the regression class that took PREFER_AOT dark had + exactly one detector: a pod. `tests/harness/hub_env.py` models the hub's + resolution rule (an entry reaches the pod only if the release DECLARES its + name, plus the reserved-namespace defence) and reports withholdings in + th#1650's typed vocabulary. `--hub-env` boots the mint child through it, + **stripping ambient values** so a developer's own shell cannot stand in for a + hub-delivered one — that substitution is what the blind spot was made of, and a + mode that allowed it would be decoration. `tests/test_hub_env_delivery_pgw995.py` + drives the real `config.load_settings()` over hub-resolved environments and + reproduces the postmortem in milliseconds: release N declares the name and the + value reaches `Settings`; release N+1 is rebuilt without the declaration, + nobody touches the entry, and the value is withheld **and said so**. Full + HelloAck-shaped boot (Vault, `applies_to` version/tag matching, the mTLS + resolve path) is filed on pgw#995 with an owner. Hub half: **th#1650**. diff --git a/scripts/config_reads_allowlist.txt b/scripts/config_reads_allowlist.txt index 218bbdfc..0bd8e464 100644 --- a/scripts/config_reads_allowlist.txt +++ b/scripts/config_reads_allowlist.txt @@ -139,10 +139,7 @@ src/gen_worker/models/svdq.py::GEN_WORKER_SVDQ_ENGINE STANDALONE GEN_WORKER_SVDQ src/gen_worker/aot_resume.py::GEN_WORKER_MINT_RESUME_MAX_BYTES STANDALONE GEN_WORKER_MINT_RESUME_MAX_BYTES, resume-area capacity bound. src/gen_worker/aot_resume.py::GEN_WORKER_MINT_RESUME_DIR STANDALONE GEN_WORKER_MINT_RESUME_DIR. pgw#929: the ENABLE used to be hidden inside an empty string; `resume_enabled()` now states it. src/gen_worker/mint_delegate.py::GEN_WORKER_MINT_IN_PROCESS STANDALONE GEN_WORKER_MINT_IN_PROCESS forces the in-process mint shape, which VIOLATES the liveness contract and is kept reachable to prove that, not to run. -src/gen_worker/mint_delegate.py::GEN_WORKER_EAGER_FIRST_BOOT STANDALONE GEN_WORKER_EAGER_FIRST_BOOT, second reader; the two switches move together. -src/gen_worker/executor.py::GEN_WORKER_BG_YIELD STANDALONE GEN_WORKER_BG_YIELD=0 restores the pre-pgw#677 shape. See the kill-switch note below. -src/gen_worker/executor.py::GEN_WORKER_EAGER_FIRST_BOOT STANDALONE GEN_WORKER_EAGER_FIRST_BOOT=0. See the kill-switch note below. # --------------------------------------------------------------------------- # TRIPWIRE — guards whose entire purpose is to fire on a misconfiguration. # --------------------------------------------------------------------------- @@ -150,42 +147,65 @@ src/gen_worker/content_credentials.py::$env_name TRIPWIRE GEN_WORKER_C2PA_KEY_PE src/gen_worker/models/memory.py::GEN_WORKER_FORBID_CPU_OFFLOAD TRIPWIRE GEN_WORKER_FORBID_CPU_OFFLOAD, now enforced at the REAL placement boundary (pgw#929 AMBIGUOUS #1). Read as env rather than Settings because a control-plane box exports it box-wide with no worker config in sight. src/gen_worker/benchmarks/swap_latency.py::GEN_WORKER_FORBID_CPU_OFFLOAD TRIPWIRE GEN_WORKER_FORBID_CPU_OFFLOAD, the original single reader. Retained: refusing the benchmark is still correct. src/gen_worker/host_move_guard.py::GEN_WORKER_HOST_MOVE_GUARD TRIPWIRE GEN_WORKER_HOST_MOVE_GUARD, the pgw#763 host-move guard's disable escape hatch. See the kill-switch note below. -src/gen_worker/aot_wrapper_split.py::GEN_WORKER_AOT_WRAPPER_SPLIT_OFF TRIPWIRE GEN_WORKER_AOT_WRAPPER_SPLIT_OFF, v1 ctor-split disable. See the kill-switch note below. src/gen_worker/aot_mint.py::PODGUARD_STATE TRIPWIRE External watchdog adapter with a producer OUTSIDE this process (`podguard.arm()` on rented pods), so there is no argv to move it to and no Settings that could own it. Gated on podguard_status()==armed; pgw#929 owns its validation and observability permanently and deletion is explicitly out of scope. # --------------------------------------------------------------------------- -# NOTE — the seven "kill switches" pgw#929 lists for deletion. FIVE can go once -# their unconditional path has a named observable; TWO CANNOT, and saying so is -# part of the finding. +# NOTE — the BEHAVIOUR axis (pgw#995), and the kill-switch verdicts. # -# Measured on the standing hub 2026-08-03: `release_env_declarations` and -# `endpoint_env_entries` carry exactly two GEN_WORKER_* names fleet-wide, -# GEN_WORKER_PREFER_AOT (19 declarations, 2 live entries) and -# GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF (5 declarations, 1 live entry). None of the -# seven kill switches is declared anywhere. +# The six classifications above answer WHERE a read happens relative to the +# config pipeline. They do NOT answer whether the read SELECTS BEHAVIOUR, and +# those are orthogonal. `GEN_WORKER_PREFER_AOT` was a behaviour switch that +# silently disarmed on a release rebuild and took the whole AOT path dark for +# three pod attempts; nothing in a WHERE-shaped classification could have +# flagged it. Evidence the axis was missing rather than implicit: this file used +# to classify `GEN_WORKER_AOT_EXPORT_PARALLEL` and `_REUSE` as LIBRARY (torch has +# never heard of either name — they are first-party feature gates) and +# `GEN_WORKER_BG_YIELD` / `_EAGER_FIRST_BOOT` / `_MINT_IN_PROCESS` as STANDALONE +# ("a CLI that loads no app config") while all three were read from executor.py +# on the serving hot path of every production pod. # -# Zero declarations is NOT the same fact as safe-to-delete, and the direction of -# the default decides which: +# So: `scripts/lint_config_reads.py` now also asserts the behaviour axis, and +# Paul's rule is enforced rather than restated — env carries CONFIG, SECRETS and +# TUNING VALUES. A branch selector needs typed config, a loud typed observable, +# and a named threat. # -# DEFAULT-ON (HOST_MOVE_GUARD, MINT_IN_PROCESS, EAGER_FIRST_BOOT, BG_YIELD, -# AOT_WRAPPER_SPLIT_OFF) — deleting the env makes the CURRENT -# PRODUCTION BEHAVIOUR unconditional. Safe. Blocked only on the -# replacement observable pgw#929 requires, plus the fact that the -# tests set BG_YIELD=0 / EAGER_FIRST_BOOT=0 as red-verification -# seams: those need a real parameter, not an env, before the -# switch goes. +# THE DIRECTION OF THE DEFAULT DECIDES WHETHER A GATE CAN BE DELETED: # -# DEFAULT-OFF (AOT_EXPORT_PARALLEL, AOT_EXPORT_REUSE) — deleting the env would -# make an UNPROVEN path unconditional. Both ship dark by their own -# docstrings ("turning it on needs the export-phase VRAM -# measurement first"; "ships dark until a real family's mint has -# run the gate on a pod"), and neither enable path has ever run in -# production. Making them unconditional is not a deletion, it is a -# launch. pgw#929's "delete seven kill switches" is WRONG for -# these two. +# DEFAULT-ON — deleting the env makes the CURRENT PRODUCTION BEHAVIOUR +# unconditional, so no pod changes behaviour. Safe. +# DEFAULT-OFF — deleting the env makes an UNPROVEN path unconditional. That +# is not a deletion, it is a LAUNCH. pgw#929's "delete seven kill +# switches" is WRONG for these, and saying so is the finding. # -# If they encode a decision nobody makes — which the measurement -# supports — the §2.2 answer is to delete the FEATURES, ~500 lines -# of dark code. That is a call for Paul, not a side effect of an -# env census, so it is recorded here rather than taken. +# ...but only together with a MEASURED declaration count. Zero fleet +# declarations is what licenses a deletion, and it is a fact you query, not one +# you infer from the code — `GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF` looks identical +# to its deleted sibling in this file and is LIVE on the standing hub. +# +# VERDICTS (declaration counts measured on the standing hub 2026-08-03): +# +# DELETED by pgw#995 — default-ON, zero declarations: +# GEN_WORKER_BG_YIELD pgw#677 bg-yield is now unconditional +# GEN_WORKER_EAGER_FIRST_BOOT pgw#671 eager-first is now unconditional +# GEN_WORKER_AOT_WRAPPER_SPLIT_OFF v1 ctor-split is now unconditional +# +# NOT DELETED, and each for a stated reason: +# GEN_WORKER_MINT_IN_PROCESS default-ON, zero declarations, SAFE to +# delete — blocked only on ten test sites across seven files that +# force the shape via this env and drive the executor. A strictly +# better parameter seam already exists (`enable_compiled(delegate= +# False)` -> `caller_forced_in_process`). Owner on pgw#995. +# GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF LIVE: 5 SDXL releases declare it, 1 +# endpoint carries a non-deleted entry. Deleting it changes a +# running endpoint. +# GEN_WORKER_AOT_EXPORT_PARALLEL default-OFF and dark by its own +# GEN_WORKER_AOT_EXPORT_REUSE docstring. Deleting the gate LAUNCHES an +# unproven path. If they encode a decision nobody makes, the §2.2 +# answer is to delete the ~500 lines of FEATURE — a call for Paul, +# not a side effect of an env census. +# GEN_WORKER_HOST_MOVE_GUARD ruled exception: safety guard, on by +# default, named threat, documented. Do not touch. +# GEN_WORKER_PROBE / _PUBLISH_ARMED security boundary; deliberately NOT a +# Settings field so tenant-adjacent code cannot reach it. +# PODGUARD_STATE producer is outside this process. # --------------------------------------------------------------------------- diff --git a/scripts/lint_config_reads.py b/scripts/lint_config_reads.py index bec74c95..a81de34f 100644 --- a/scripts/lint_config_reads.py +++ b/scripts/lint_config_reads.py @@ -221,6 +221,207 @@ def load_allowlist() -> Tuple[Dict[Tuple[str, str], str], List[str]]: return allowed, errors +# --------------------------------------------------------------------------- +# pgw#995 — the BEHAVIOUR axis +# --------------------------------------------------------------------------- +# +# The six classifications above answer WHERE a read happens relative to the +# config pipeline. None of them answers whether the read SELECTS BEHAVIOUR, and +# those are orthogonal questions. `GEN_WORKER_PREFER_AOT` was a behaviour switch +# that silently disarmed on a release rebuild and took the entire AOT path dark +# for three pod attempts; nothing WHERE-shaped could have flagged it. The +# allowlist proves the axis was missing rather than implicit: it classified +# `GEN_WORKER_AOT_EXPORT_PARALLEL`/`_REUSE` as LIBRARY (torch has never heard of +# either name) and three serving-hot-path switches as STANDALONE ("a CLI that +# loads no app config"). +# +# Paul's rule: env carries CONFIG, SECRETS and TUNING VALUES. A branch selector +# needs typed config, a loud typed observable, and a named threat. This gate +# makes a NEW one fail the build instead of being noticed three pods later. + +#: (path, ENV_NAME) -> the named threat this gate defends against. +#: A gate lives here ONLY with a threat a reader can evaluate. "It is useful" +#: and "it is off by default" are not threats. +BEHAVIOUR_GATES: Dict[Tuple[str, str], str] = { + ("src/gen_worker/host_move_guard.py", "GEN_WORKER_HOST_MOVE_GUARD"): + "RULED EXCEPTION. Safety guard, ON by default, disabled only with =0. " + "Threat: a silent host-RAM offload turns a serving pod into a " + "swap-thrashing one that still answers health checks. Documented in " + "CLAUDE.md; explicitly out of scope for every env sweep.", + ("src/gen_worker/procsplit/actions.py", "GEN_WORKER_PROBE"): + "SECURITY BOUNDARY (pgw#980). Marks the pod a live-edit probe and " + "DISARMS cell publish in the parent's action allowlist. Threat: a probe " + "pod publishing a cell minted from hand-edited source into the fleet " + "store. Deliberately NOT a Settings field: `authorize` is the boundary, " + "and a guard that depends on a config load having succeeded has an " + "unarmed window.", + ("src/gen_worker/procsplit/actions.py", "GEN_WORKER_PROBE_PUBLISH_ARMED"): + "SECURITY BOUNDARY (pgw#980). The separate second decision that re-arms " + "publish on a marked probe. Two names so 'this is a probe' and 'this " + "probe may write' can never be satisfied by one value.", + ("src/gen_worker/procsplit/__init__.py", "GEN_WORKER_COMPUTE_CHILD"): + "BOOTSTRAP. Decides WHICH OF TWO PROGRAMS this process is, before " + "_run_main and before any config can exist. Threat: none — it is not a " + "policy, it is the process's own identity, and there is no earlier " + "carrier than the environment it was exec'd with.", + ("src/gen_worker/supervisor.py", "GEN_WORKER_SUPERVISOR"): + "BOOTSTRAP. Pre-fork supervisor predicate, runs before the process " + "entry. Same identity-not-policy reasoning.", + ("src/gen_worker/supervisor.py", "GEN_WORKER_SUPERVISED"): + "BOOTSTRAP. Pre-fork re-entry guard; without it the supervisor forks " + "itself forever.", + ("src/gen_worker/models/memory.py", "GEN_WORKER_FORBID_CPU_OFFLOAD"): + "TRIPWIRE at the real placement boundary. Read as env rather than " + "Settings because a control-plane box exports it box-wide with no worker " + "config in sight. Threat: a CPU-offloading run on the shared dev box.", + ("src/gen_worker/benchmarks/swap_latency.py", "GEN_WORKER_FORBID_CPU_OFFLOAD"): + "TRIPWIRE, the original single reader. Refusing the benchmark is still " + "correct. Same threat, same box-wide-export reasoning.", + ("src/gen_worker/mint_delegate.py", "GEN_WORKER_MINT_IN_PROCESS"): + "DEFECT, NOT AN EXCEPTION — listed so the gate is green while it is " + "burned down, exactly as VIOLATION lines are. Default-ON, zero fleet " + "declarations, and `enable_compiled(delegate=False)` is a strictly " + "better parameter seam that already exists. Blocked ONLY on ten test " + "sites across seven files that force the shape via this env and drive " + "the executor. Owner and scope on pgw#995. Do not add a second reader.", + ("src/gen_worker/aot_export_parallel.py", "GEN_WORKER_AOT_EXPORT_PARALLEL"): + "DARK FEATURE, default OFF. NOT deletable: deleting the gate would make " + "an unproven path unconditional, which is a LAUNCH, not a deletion. " + "Threat: an unmeasured export-phase VRAM footprint OOMs a 74-minute " + "phase. `decide()` emits the decision on EVERY mint whether or not the " + "flag is on, so the gate's state is observable rather than silent — " + "which is the property PREFER_AOT lacked.", + ("src/gen_worker/aot_export_reuse.py", "GEN_WORKER_AOT_EXPORT_REUSE"): + "DARK FEATURE, default OFF. Same launch-not-deletion reasoning. Threat: " + "a reused export whose artifact is not byte-identical silently changes " + "a compiled program. The gate byte-compares every emitted file and " + "falls back on any doubt (ReuseUnproven).", + ("src/gen_worker/aot_wrapper_split.py", "GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF"): + "LIVE ON THE FLEET, and that is why it survives its deleted sibling: 5 " + "SDXL releases declare it and 1 endpoint carries a non-deleted entry " + "(standing hub, 2026-08-03). Deleting a live switch changes a running " + "endpoint. Threat: the pgw#811 run_impl split regressing a family, with " + "no way to unstick it short of a release.", + ("src/gen_worker/lifecycle.py", "$ENV_VAR"): + "READ-ONLY WARNING PREDICATE. Re-reads the hub-delivered topology only " + "to decide whether the 'GPUs are invisible' warning applies. Selects a " + "log line, never a code path.", + ("src/gen_worker/executor.py", "RUNPOD_POD_ID"): + "DEFECT, listed to keep the gate green while it burns down. A vendor env " + "used as a proxy for 'managed runtime'. Blocked on pgw#921/th#1488 " + "RuntimeIdentity.managed; pgw#929 AMBIGUOUS #5 forbids papering over it " + "with a vendor Settings field.", + ("src/gen_worker/executor.py", "RUNPOD_PROVIDER"): + "DEFECT, same site and same blocker as RUNPOD_POD_ID above.", + ("src/gen_worker/content_credentials.py", "$env_name"): + "TRIPWIRE that REFUSES THE BOOT (th#1307). Threat: a C2PA private key " + "reaching a pod. Carries no behaviour of its own — its only outcome is " + "a loud refusal.", + +} + +#: Predicate-shaped function names: a `return ` inside one of these is +#: a behaviour selection even without a syntactic `if`. +_PREDICATE_SUFFIXES = ("enabled", "disabled", "_on", "_off", "armed", "forced") + + +class BehaviourVisitor(ast.NodeVisitor): + """Env reads whose value reaches a CONDITIONAL rather than a value slot. + + Deliberately syntactic and conservative. It cannot follow a read through a + variable into an `if` three functions away, and it does not try — a gate + that pretends to completeness it does not have is worse than one whose + reach is stated. What it DOES catch is every shape the four switches + deleted by pgw#995 were written in, which is the shape this defect keeps + being written in. + """ + + def __init__(self) -> None: + self.hits: List[Tuple[int, str]] = [] + + def _collect(self, node: ast.AST) -> None: + sub = EnvVisitor() + sub.visit(node) + self.hits.extend(sub.hits) + + def _scan_conditions(self, tree: ast.AST, consts: Dict[str, str]) -> None: + for node in ast.walk(tree): + if isinstance(node, (ast.If, ast.While, ast.IfExp, ast.Assert)): + self._collect_with(node.test, consts) + elif isinstance(node, ast.comprehension): + for cond in node.ifs: + self._collect_with(cond, consts) + elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + returns_bool = ( + isinstance(node.returns, ast.Name) + and node.returns.id == "bool") + predicate_name = node.name.lower().endswith(_PREDICATE_SUFFIXES) + if not (returns_bool or predicate_name): + continue + for inner in ast.walk(node): + if isinstance(inner, ast.Return) and inner.value is not None: + self._collect_with(inner.value, consts) + + def _collect_with(self, node: ast.AST, consts: Dict[str, str]) -> None: + sub = EnvVisitor() + sub.consts = dict(consts) + sub.visit(node) + self.hits.extend(sub.hits) + + +def scan_behaviour() -> Dict[Tuple[str, str], int]: + """Every env read outside `config/` that feeds a conditional.""" + sites: Dict[Tuple[str, str], int] = {} + for path in sorted(SRC_ROOT.rglob("*.py")): + if CONFIG_PKG in path.parents or path == CONFIG_PKG: + continue + try: + tree = ast.parse(path.read_text(encoding="utf-8")) + except SyntaxError: # pragma: no cover - the other pass reports it + continue + consts = EnvVisitor() + consts.load_consts(tree) + visitor = BehaviourVisitor() + visitor._scan_conditions(tree, consts.consts) + rel = str(path.relative_to(REPO)) + for lineno, name in visitor.hits: + sites.setdefault((rel, name or UNRESOLVED), lineno) + return sites + + +def check_behaviour_gates() -> List[str]: + """Paul's rule, enforced: env carries values, never a branch selection.""" + found = scan_behaviour() + errors: List[str] = [] + for key in sorted(set(found) - set(BEHAVIOUR_GATES)): + path, name = key + errors.append( + f"{path}:{found[key]} reads {name} from the environment and feeds it " + f"to a CONDITIONAL. Env vars are for CONFIG and SECRETS, never logic " + f"or behaviour switches (Paul, standing rule). GEN_WORKER_PREFER_AOT " + f"was exactly this: it gated the mint recipe and cell discovery, " + f"silently disarmed on a release rebuild, and cost three pod attempts " + f"before anyone noticed the AOT path was dark.\n" + f" Fix it: make the branch unconditional (if its default is ON and " + f"nothing declares it, deleting the gate changes NO pod's behaviour), " + f"or move the decision to typed config with a loud typed observable.\n" + f" If it genuinely must stay, add ('{path}', '{name}') to " + f"BEHAVIOUR_GATES in {Path(__file__).name} with the THREAT it defends " + f"against — not 'it is useful' and not 'it is off by default'.") + for key in sorted(set(BEHAVIOUR_GATES) - set(found)): + path, name = key + errors.append( + f"BEHAVIOUR_GATES lists ('{path}', '{name}') but no such conditional " + f"env read exists any more. Delete the entry — a stale exemption is " + f"the second carrier this whole gate exists to prevent (§4.22).") + for key, threat in BEHAVIOUR_GATES.items(): + if len(threat.strip()) < 40: + errors.append( + f"BEHAVIOUR_GATES{key}: the threat must be specific enough for a " + f"reader to evaluate, got {threat!r}") + return errors + + def check_owned_names_known(names: Set[str]) -> List[str]: """Every owned-namespace name read in src/ must be known to the loader.""" sys.path.insert(0, str(REPO / "src")) @@ -246,6 +447,7 @@ def check_owned_names_known(names: Set[str]) -> List[str]: def main() -> int: sites, names = scan() allowed, errors = load_allowlist() + errors.extend(check_behaviour_gates()) for key in sorted(set(sites) - set(allowed)): path, name = key diff --git a/scripts/micro_mint_rig.py b/scripts/micro_mint_rig.py index 594d21fb..749bde6f 100755 --- a/scripts/micro_mint_rig.py +++ b/scripts/micro_mint_rig.py @@ -38,6 +38,24 @@ * `nice`, so a compile cannot starve the box's other agents; * a load gate: refuse to start above 1-min load 24; * `GEN_WORKER_HOST_MOVE_GUARD` untouched — the rig never disables it. + +THE ENV-DELIVERY MODE (pgw#995, `--hub-env`). The rig above builds the child's +environment itself — `mint_process.child_env` plus a few rig keys — which is a +shape no production pod ever has. So the chain that actually delivers env to a +pod was invisible to it, and to every other test in this repo: + + worker function declares env -> release_env_declarations + operator sets a value -> endpoint_env_entries + pod launch -> EndpointEnvService.Resolve -> pod env -> Settings + +That chain is what took `GEN_WORKER_PREFER_AOT` dark: a release rebuild stopped +declaring the name, the hub withheld the live entry SILENTLY, and three pod +attempts went by. `--hub-env` boots the mint child through it instead — the +child's environment is what the hub's rule would have delivered, ambient values +are STRIPPED so a developer's shell cannot stand in for a delivered one, and any +withholding is reported as a rig fact rather than a missing variable nobody +looks for. The model lives in `tests/harness/hub_env.py`; the first regression +tests are `tests/test_hub_env_delivery_pgw995.py`. """ from __future__ import annotations @@ -272,9 +290,39 @@ def _mint_request( task, workdir=workdir, cap_bytes=cap_bytes) +#: What the rig's endpoint function DECLARES, the way a real build reads it off +#: the function schema. Deliberately tiny: the point is the delivery mechanism, +#: not the breadth of the catalogue. +RIG_DECLARED_ENV = ("HF_TOKEN",) + +#: Ambient names the rig REFUSES to let through in `--hub-env` mode. Without +#: this the developer's own shell satisfies the assertion and the mode proves +#: nothing — which is the exact substitution the blind spot was made of. +RIG_STRIPPED_ENV = ("HF_TOKEN",) + + +def hub_delivered_env( + base: Dict[str, str], entries: Optional[Dict[str, str]] = None, +) -> tuple: + """(env, withheld) as the hub would resolve them for this rig's release.""" + sys.path.insert(0, str(REPO / "tests")) + from harness import hub_env as _hub + + delivery = _hub.resolve( + _hub.declared_by(list(RIG_DECLARED_ENV)), + _hub.EndpointEnvEntries(dict(entries or {}))) + env = _hub.pod_environ(base, delivery, strip=RIG_STRIPPED_ENV) + return env, [ + {"name": w.name, "reason": w.reason, "detail": w.detail} + for w in delivery.withheld + ] + + def run_cycle( root: Path, *, stage: str = "all", force_load: bool = False, device: str = "auto", vehicle: str = DEFAULT_VEHICLE, + hub_env_mode: bool = False, + hub_env_entries: Optional[Dict[str, str]] = None, ) -> RigResult: from harness import rig_vehicles @@ -346,6 +394,13 @@ def run_cycle( g0 = time.monotonic() phases: List[str] = [] env = dict(mp.child_env(request)) + hub_withheld: List[Dict[str, str]] = [] + if hub_env_mode: + # pgw#995: the child no longer inherits whatever this process carries. + # It boots with what the HUB would have delivered for this release — + # so a name the release stops declaring disappears here, locally, the + # way it disappeared on the pod nobody was watching. + env, hub_withheld = hub_delivered_env(env, hub_env_entries) env["PYTHONPATH"] = os.pathsep.join( [str(REPO / "tests"), str(REPO / "src"), *veh.syspath, env.get("PYTHONPATH", "")]) @@ -364,6 +419,8 @@ def run_cycle( leg.facts = { "status": outcome.status, "exit_code": outcome.exit_code, + "hub_env_mode": hub_env_mode, + "hub_env_withheld": hub_withheld, "phases_seen": sorted(set(phases)), "phase_seconds": dict(report.phases) if report else {}, "peak_vram_bytes": int(report.peak_vram_bytes) if report else 0, @@ -572,6 +629,15 @@ def main(argv: Optional[List[str]] = None) -> int: help="WHAT to mint: 'tiny' (pgw#978's one-entry plumbing toy) or " "'micro' (pgw#997's org-worker-shaped examples/micro-diffusion — " "3 entries, container inputs, generated weights)") + parser.add_argument( + "--hub-env", action="store_true", + help="boot the mint child through the hub's env-delivery rule " + "(declarations x entries) instead of this process's environment; " + "ambient values are stripped and withholdings are reported") + parser.add_argument( + "--hub-env-entry", action="append", default=[], metavar="NAME=VALUE", + help="an endpoint_env_entries row the operator has set; repeatable. " + "Only names the release DECLARES are delivered.") args = parser.parse_args(list(argv) if argv is not None else None) root = Path(args.root) @@ -580,8 +646,16 @@ def main(argv: Optional[List[str]] = None) -> int: root.mkdir(parents=True, exist_ok=True) try: + entries: Dict[str, str] = {} + for raw in args.hub_env_entry: + name, sep, value = str(raw).partition("=") + if not sep: + parser.error(f"--hub-env-entry expects NAME=VALUE, got {raw!r}") + entries[name.strip()] = value result = run_cycle(root, stage=args.stage, force_load=args.force_load, - device=args.device, vehicle=args.vehicle) + device=args.device, vehicle=args.vehicle, + hub_env_mode=args.hub_env, + hub_env_entries=entries) except RigRefused as exc: print(f"REFUSED: {exc}", file=sys.stderr) return 2 diff --git a/src/gen_worker/aot_wrapper_split.py b/src/gen_worker/aot_wrapper_split.py index 80d23488..cd39334b 100644 --- a/src/gen_worker/aot_wrapper_split.py +++ b/src/gen_worker/aot_wrapper_split.py @@ -296,13 +296,22 @@ def _reinline(source: str) -> str: # Mint-path installation # --------------------------------------------------------------------------- -#: Env kill-switch. Not a sealed config knob and not an inductor config — -#: see :func:`install` for why that distinction is what keeps cell identity -#: untouched. Kills the ctor and run_impl levers together. -DISABLE_ENV = "GEN_WORKER_AOT_WRAPPER_SPLIT_OFF" +#: pgw#995: ``GEN_WORKER_AOT_WRAPPER_SPLIT_OFF`` is GONE. It defaulted ON, no +#: release ever declared it and no endpoint ever set it, so deleting it makes +#: the shape every pod already ran unconditional. An env that selects which +#: program runs is the class that took `GEN_WORKER_PREFER_AOT` dark for three +#: pod attempts; a switch nobody has ever thrown carries none of the safety it +#: appears to and all of the silent-disarm risk. #: Kill switch for the pgw#811 ``run_impl`` split alone, so the (much older, #: much better travelled) ctor split can stay on if run_impl ever goes off. +#: pgw#995 KEEPS this one, and the asymmetry with its deleted sibling above is +#: the finding rather than an oversight: this name is LIVE — five SDXL releases +#: (0.2.111/112/113/116/117) declare it and one endpoint carries a non-deleted +#: `endpoint_env_entries` row (verified on the standing hub 2026-08-03). Deleting +#: a live switch changes a running endpoint's behaviour; deleting a never-thrown +#: one cannot. "Zero declarations" is the fact that licenses a deletion, and it +#: is a fact you MEASURE, not one you assume from the code. DISABLE_V2_ENV = "GEN_WORKER_AOT_RUN_IMPL_SPLIT_OFF" #: How many of the K+1 part compiles may run at once. pgw#809's pool owns @@ -526,14 +535,11 @@ def install() -> bool: ``code_closure`` axis is untouched too. No cell is re-keyed. (Both facts are asserted by tests/test_aot_wrapper_split_pgw793.py.) - Returns True when installed, False when already installed or disabled. + Returns True when installed, False when already installed. """ global _installed if _installed: return False - if os.environ.get(DISABLE_ENV, "").strip(): - logger.info("aot-wrapper-split: disabled by %s", DISABLE_ENV) - return False try: from torch._inductor import cpp_builder except Exception: @@ -632,7 +638,6 @@ def _patched(cmd_line: str, cwd: str) -> None: "CHUNK", "EVENT", "MIN_STATEMENTS", - "DISABLE_ENV", "DISABLE_V2_ENV", "JOBS_ENV", "SERVING_HEADROOM_CPUS", diff --git a/src/gen_worker/config/loader.py b/src/gen_worker/config/loader.py index 8c7910aa..f2d02e3f 100644 --- a/src/gen_worker/config/loader.py +++ b/src/gen_worker/config/loader.py @@ -138,9 +138,6 @@ "GEN_WORKER_INTERNAL_OBJECT_HOSTS", "GEN_WORKER_AOT_EXPORT_PARALLEL", "GEN_WORKER_AOT_EXPORT_REUSE", - "GEN_WORKER_AOT_WRAPPER_SPLIT_OFF", - "GEN_WORKER_BG_YIELD", - "GEN_WORKER_EAGER_FIRST_BOOT", "GEN_WORKER_MINT_IN_PROCESS", "GEN_WORKER_HOST_MOVE_GUARD", "GEN_WORKER_FORBID_CPU_OFFLOAD", diff --git a/src/gen_worker/executor.py b/src/gen_worker/executor.py index f71c3e35..1006b5da 100644 --- a/src/gen_worker/executor.py +++ b/src/gen_worker/executor.py @@ -655,14 +655,6 @@ def _snapshot_to_resolved(snap: pb.Snapshot) -> "WorkerResolvedRepo": _BG_THREAD_ADMIT_WAIT_S = 0.5 -def _bg_yield_enabled() -> bool: - """pgw#677: default ON; env kill switch for red-verification and - emergencies only. OFF restores the pre-fix shape: mint seeds idle-gate - + hold the bare run gate (inline compiles included) and shape-warm - compiles run ungated against tenant forwards.""" - return os.environ.get("GEN_WORKER_BG_YIELD", "1").strip() != "0" - - def _cell_execution_lane_matches( ref: str, family: str, @@ -2884,11 +2876,6 @@ def _delegated_pendings(pendings: typing.Mapping[int, Any]) -> bool: return any(getattr(p, "delegated", False) for p in pendings.values()) -def _eager_first_boot_enabled() -> bool: - """pgw#671: default ON; env kill switch for emergencies only.""" - return os.environ.get("GEN_WORKER_EAGER_FIRST_BOOT", "1").strip() != "0" - - @dataclass class _HostRamBlock: """One exact, still-unsatisfied host-RAM admission observation.""" @@ -9222,8 +9209,6 @@ def _eager_first_eligible( lane. A delegated pending's eager tier is the untouched pipeline itself; the router question belongs only to an in-process capture, whose eager-while-compiling routing is what a router performs.""" - if not _eager_first_boot_enabled(): - return False if not inj.pending_self_mints: return False if spec.cls is not None and callable(getattr(spec.cls, "warmup", None)): @@ -9706,22 +9691,6 @@ async def _unit(wj: Any) -> bool: in their own turns. False = preempted by a tenant arrival; the caller re-queues the unit.""" _checkpoint() - if not _bg_yield_enabled(): - # Legacy shape (kill switch / red-verification): idle-gate - # between units, bare run gate around the forward. - idle = asyncio.ensure_future(self.wait_idle()) - stop = asyncio.ensure_future(bg.abandon.wait()) - try: - await asyncio.wait( - {idle, stop}, return_when=asyncio.FIRST_COMPLETED) - finally: - for fut in (idle, stop): - if not fut.done(): - fut.cancel() - _checkpoint() - async with rec.run_lock: - await _forward(wj) - return True async with self._bg_turn(rec, "seed", abort=bg.abandon) as stole: _checkpoint() try: @@ -11534,10 +11503,11 @@ def abort_check() -> None: def _wire_turn_gate(self, rec: _ClassRecord, pipeline: Any) -> None: """Hand this pipeline's hot-swap router the background-turn gate so every shape-warm/heal compile serializes with — and yields to — - tenant work (pgw#677). Idempotent; no-op without a router.""" - if not _bg_yield_enabled(): - return + tenant work (pgw#677). Idempotent; no-op without a router. + pgw#995: unconditional. ``GEN_WORKER_BG_YIELD`` used to be able to skip + this, restoring the pre-pgw#677 shape where shape-warm compiles ran + ungated against tenant forwards. Nothing ever set it.""" router = hot_swap.router_of(pipeline) if router is not None: router.set_turn_gate(self._bg_turn_threaded(rec)) diff --git a/src/gen_worker/fleet_cells.py b/src/gen_worker/fleet_cells.py index 678d2eca..fe10c3b7 100644 --- a/src/gen_worker/fleet_cells.py +++ b/src/gen_worker/fleet_cells.py @@ -1851,9 +1851,10 @@ def _unregister(pending: "PendingSelfMint") -> None: #: off") that named two causes which were BOTH false on the measured pod while #: the true cause — the pipeline-side mandatory-lane misclassification — was #: not named at all. A refusal that cannot name its own cause is the defect. +#: pgw#995 dropped `eager_first_disabled`: eager-first is unconditional, so +#: that cause can no longer arise and a reason nobody can reach is dead prose. _DELEGATION_DECLINE_PHASE = { "mint_in_process_forced": "aot_mint_forced_in_process", - "eager_first_disabled": "aot_eager_first_disabled", "no_eager_tier": "aot_no_eager_tier", "caller_forced_in_process": "aot_mint_forced_in_process", } @@ -1862,9 +1863,6 @@ def _unregister(pending: "PendingSelfMint") -> None: "GEN_WORKER_MINT_IN_PROCESS is set, which forces the in-process " "capture; an AOTI export has no eager tier to serve from while it " "compiles, so it cannot ride that shape", - "eager_first_disabled": - "GEN_WORKER_EAGER_FIRST_BOOT=0 turned eager-first off, and delegation " - "IS eager-first — there is no route to serve while a child compiles", "no_eager_tier": "an armed non-eager backend (AOTI cell or TRT engine) has replaced " "this pipeline's forward, so there is no eager tier to serve from", diff --git a/src/gen_worker/mint_delegate.py b/src/gen_worker/mint_delegate.py index 5d1656a0..42a6a33b 100644 --- a/src/gen_worker/mint_delegate.py +++ b/src/gen_worker/mint_delegate.py @@ -48,32 +48,41 @@ #: Kill switch, for red-verifying the in-process shape only. Delegation is the #: default because the in-process shape VIOLATES the liveness contract #: (WORKER-CONTRACTS §1/§2) — it is kept reachable to prove that, not to run. +#: +#: pgw#995: this is the LAST surviving env behaviour switch in this repo whose +#: default is ON and whose fleet declaration count is zero — i.e. the last one +#: that is safe to delete — and it is not deleted here. The reason is specific, +#: not a shrug: ``fleet_cells.enable_compiled`` ALREADY takes ``delegate=False`` +#: and reports it as ``caller_forced_in_process``, so a strictly better +#: parameter seam exists and this env is redundant with it. What blocks the +#: deletion is that ten test sites across seven files force the shape by setting +#: this env and then driving the EXECUTOR, which reaches the policy with +#: ``delegate=None``; threading the parameter through those call sites is a real +#: refactor of a 4k-line test file and does not belong in a sweep. Owner and +#: scope are recorded on pgw#995. Do not add a second reader in the meantime. ENV_IN_PROCESS = "GEN_WORKER_MINT_IN_PROCESS" #: Typed refusals :func:`delegation_refusal` can return. They are the OPERATOR -#: half of the decision (env kill switches); the PIPELINE half lives in +#: half of the decision; the PIPELINE half lives in #: ``fleet_cells.delegation_refusal``. pgw#813: the two were collapsed into one #: either/or sentence on the wire, so a real refusal named two causes that were #: both false and never named the one that was true. +#: +#: pgw#995 deleted ``REFUSAL_EAGER_FIRST_DISABLED`` with its switch. Delegation +#: IS eager-first, so the two moved together — and once eager-first is +#: unconditional, "eager-first is off" is not a state this worker can be in. +#: A refusal reason that can never be returned is a cause a reader will hunt for +#: and never find, which is the same defect as a cause that goes unnamed. REFUSAL_IN_PROCESS_FORCED = "mint_in_process_forced" -REFUSAL_EAGER_FIRST_DISABLED = "eager_first_disabled" def delegation_refusal() -> str: - """"" when this WORKER may mint out of process, else the typed reason. - - Delegation IS eager-first: the live pipeline is never armed, so a boot that - has eager-first turned off has no route to serve while a child compiles. - The two switches therefore move together rather than combining into a - fourth, undefined shape. - """ + """"" when this WORKER may mint out of process, else the typed reason.""" if os.environ.get(ENV_IN_PROCESS, "").strip().lower() in ( "1", "true", "yes", "on" ): return REFUSAL_IN_PROCESS_FORCED - if os.environ.get("GEN_WORKER_EAGER_FIRST_BOOT", "1").strip() == "0": - return REFUSAL_EAGER_FIRST_DISABLED return "" diff --git a/tests/harness/hub_env.py b/tests/harness/hub_env.py new file mode 100644 index 00000000..5e28880f --- /dev/null +++ b/tests/harness/hub_env.py @@ -0,0 +1,155 @@ +"""pgw#995 — the hub's env-delivery chain, modelled so local tests can see it. + +THE BLIND SPOT THIS CLOSES. `scripts/micro_mint_rig.py` (pgw#978) runs the whole +mint machinery on this box and is why a change can be proven before PyPI. But it +**constructs its own environment**: the mint child gets `mint_process.child_env` +plus a few rig keys, and the adopting process gets `dict(os.environ)`. Neither +resembles how a production pod is given its env, so the chain below was +invisible to every test in this repo: + + worker function declares env -> build schema -> release_env_declarations + operator sets a value -> endpoint_env_entries (+ Vault) + pod launch -> EndpointEnvService.Resolve -> pod env + | + config.loader ----+--> Settings + +That chain is exactly what took `GEN_WORKER_PREFER_AOT` dark. The flag was +declared by the worker function and set on the endpoint; a release rebuild +stopped declaring the name, the hub withheld every matching entry **silently**, +and three pod attempts went by before anyone noticed the AOT path was off. Every +component was individually correct. The DELIVERY was what broke, and delivery +was the one thing nothing tested. + +WHAT THIS IS. A faithful model of `EndpointEnvService.Resolve`'s contract — NOT +a reimplementation of the hub. It carries the one rule that matters (an entry +reaches the pod only if the release DECLARES its name) and the reserved-name +defence, and it reports withholdings with the same typed vocabulary the hub uses +(th#1650). It is deliberately small: a big fake hub would drift from the real +one and start certifying its own behaviour. + +WHAT IT IS NOT. It does not model Vault, `applies_to` version/tag matching, or +the mTLS resolve path. Those belong to the full HelloAck-shaped boot filed on +pgw#995; this is the seam plus the first regression it makes visible. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Dict, Iterable, List, Mapping, Sequence, Tuple + +#: Mirrors tensorhub `internal/api/endpoint_env_withheld.go` (th#1650). Kept +#: verbatim so a log line from a real pod and a failure from this harness use +#: the same word for the same thing. +WITHHELD_UNDECLARED = "undeclared_by_release" +WITHHELD_RESERVED = "reserved_name" + +#: Mirrors the hub's `reservedEnvPrefixes` narrowly — only what a worker test +#: can meaningfully trip. The hub is the authority; this exists so a test that +#: sets a platform name gets the platform answer rather than a false pass. +RESERVED_PREFIXES: Tuple[str, ...] = ( + "WORKER_", "TENSORHUB_", "ORCHESTRATOR_", "HF_HOME", "RUNPOD_", + "GEN_WORKER_C2PA_", "GEN_WORKER_PROCESS_SPLIT", "GEN_WORKER_COMPUTE_CHILD", + "GEN_WORKER_CHILD_", +) + + +@dataclass(frozen=True) +class Withheld: + """One entry the operator set that this pod will not receive.""" + + name: str + reason: str + detail: str = "" + + +@dataclass +class ReleaseEnvDeclarations: + """What THIS release's worker functions declare, per `release_env_declarations`. + + Per-RELEASE, which is the property that makes the postmortem possible: a + rebuild produces a new declaration set while the operator's entries are + per-ENDPOINT and long-lived. + """ + + names: Tuple[str, ...] = () + + @classmethod + def of(cls, *names: str) -> "ReleaseEnvDeclarations": + return cls(tuple(names)) + + +@dataclass +class EndpointEnvEntries: + """What the operator set, per `endpoint_env_entries`. Long-lived.""" + + values: Dict[str, str] = field(default_factory=dict) + + +@dataclass(frozen=True) +class Delivery: + """The outcome of one pod launch's env resolution.""" + + env: Dict[str, str] + withheld: Tuple[Withheld, ...] + + def withheld_names(self) -> List[str]: + return [w.name for w in self.withheld] + + +def is_reserved(name: str) -> bool: + return any(name.startswith(p) for p in RESERVED_PREFIXES) + + +def resolve( + declarations: ReleaseEnvDeclarations, + entries: EndpointEnvEntries, +) -> Delivery: + """The hub's rule: an entry reaches the pod only if the release declares it. + + Returns the delivered map AND every withholding, because a delivery that + reports only what arrived is the exact shape that hid the defect — "nothing + arrived" and "nothing was ever configured" have to be distinguishable. + """ + declared = set(declarations.names) + env: Dict[str, str] = {} + withheld: List[Withheld] = [] + for name in sorted(entries.values): + if name not in declared: + withheld.append(Withheld( + name, WITHHELD_UNDECLARED, + f"release declares {len(declared)} env name(s), not this one")) + continue + if is_reserved(name): + withheld.append(Withheld( + name, WITHHELD_RESERVED, "platform-reserved namespace")) + continue + env[name] = entries.values[name] + return Delivery(env=env, withheld=tuple(withheld)) + + +def pod_environ( + base: Mapping[str, str], + delivery: Delivery, + *, + strip: Iterable[str] = (), +) -> Dict[str, str]: + """The environment a pod actually boots with: image env + delivered entries. + + `strip` removes names the launching process happens to carry — without it a + rig running on a developer box would let an ambient export stand in for a + hub-delivered value and prove nothing. That substitution IS the blind spot; + a rig mode that allowed it would be decoration. + """ + out = {k: v for k, v in base.items() if k not in set(strip)} + out.update(delivery.env) + return out + + +def declared_by(function_env: Sequence[str]) -> ReleaseEnvDeclarations: + """Build declarations the way a build does: from the function's own list. + + Real builds read this off the function schema payload. A worker function + that stops listing a name produces a release that does not declare it — + with no error anywhere, because an empty `env` list is legal. + """ + return ReleaseEnvDeclarations(tuple(function_env)) diff --git a/tests/test_aot_mint_unblock_pgw813_pgw815.py b/tests/test_aot_mint_unblock_pgw813_pgw815.py index d62316dd..e28a3841 100644 --- a/tests/test_aot_mint_unblock_pgw813_pgw815.py +++ b/tests/test_aot_mint_unblock_pgw813_pgw815.py @@ -4,7 +4,7 @@ aot_cell_discovery miss family=sdxl lane=w8a8-lora64 self_mint_skipped aot_requires_delegation "out-of-process minting is - disabled (GEN_WORKER_MINT_IN_PROCESS or eager-first + disabled (GEN_WORKER_MINT_IN_PROCESS off) and an AOTI export has no eager tier..." self_mint_started dynamo ... armed an in-process capture @@ -272,19 +272,22 @@ def test_delegation_declines_name_their_TRUE_cause( _arm() assert "aot_mint_forced_in_process" in _phases(_events, "self_mint_skipped") + # pgw#995: the second arm here drove `GEN_WORKER_EAGER_FIRST_BOOT=0` and + # asserted the `aot_eager_first_disabled` phase. Both the switch and the + # phase are deleted — eager-first is unconditional, so that decline cannot + # arise, and a reason nobody can reach is a cause a reader hunts for and + # never finds. The pgw#813 claim under test is unharmed: it is that a + # refusal names its TRUE cause, which the operator arm above and the + # pipeline arm below still exercise. _events.clear() monkeypatch.delenv("GEN_WORKER_MINT_IN_PROCESS") - monkeypatch.setenv("GEN_WORKER_EAGER_FIRST_BOOT", "0") fleet_cells._PENDING.clear() - _arm() - assert "aot_eager_first_disabled" in _phases(_events, "self_mint_skipped") # pgw#846: `Compile.regional` is the dynamo/JIT per-block knob (ie#381) # and the AOT mint ignores it — regional EXPORT is retired, the recipe is # always whole-graph. A family that declares it must neither decline # delegation nor change the mint shape. _events.clear() - monkeypatch.delenv("GEN_WORKER_EAGER_FIRST_BOOT") fleet_cells._PENDING.clear() fleet_cells.enable_compiled( _Pipe(), _Cfg(regional=True), publisher=_Publisher()) # type: ignore[arg-type] @@ -296,11 +299,14 @@ def test_mint_delegate_names_its_own_refusals( monkeypatch: pytest.MonkeyPatch, ) -> None: monkeypatch.delenv("GEN_WORKER_MINT_IN_PROCESS", raising=False) - monkeypatch.delenv("GEN_WORKER_EAGER_FIRST_BOOT", raising=False) assert mint_delegate.delegation_refusal() == "" + # pgw#995: eager-first is unconditional, so setting the deleted name is a + # no-op rather than a second refusal. Asserted, not assumed — a deletion + # that leaves a live reader somewhere else looks exactly like this test + # passing for the wrong reason. monkeypatch.setenv("GEN_WORKER_EAGER_FIRST_BOOT", "0") - assert (mint_delegate.delegation_refusal() - == mint_delegate.REFUSAL_EAGER_FIRST_DISABLED) + assert mint_delegate.delegation_refusal() == "" + assert not hasattr(mint_delegate, "REFUSAL_EAGER_FIRST_DISABLED") monkeypatch.setenv("GEN_WORKER_MINT_IN_PROCESS", "1") assert (mint_delegate.delegation_refusal() == mint_delegate.REFUSAL_IN_PROCESS_FORCED) @@ -342,7 +348,6 @@ def test_eager_first_admits_a_DELEGATED_pending_with_no_router( lines later — pgw#784 could not run on any lane.""" from gen_worker.models import loading as loading_mod - monkeypatch.delenv("GEN_WORKER_EAGER_FIRST_BOOT", raising=False) monkeypatch.setattr( loading_mod, "pipeline_weight_lane", lambda p: "w8a8-lora64") @@ -371,7 +376,6 @@ def test_eager_first_still_requires_a_router_for_an_IN_PROCESS_capture( router, so no router means no eager tier.""" from gen_worker.models import loading as loading_mod - monkeypatch.delenv("GEN_WORKER_EAGER_FIRST_BOOT", raising=False) monkeypatch.setattr(loading_mod, "pipeline_weight_lane", lambda p: "") ex = _executor(tmp_path) diff --git a/tests/test_eager_first_boot_pgw671.py b/tests/test_eager_first_boot_pgw671.py index 5bd6a1b7..1eb9d50b 100644 --- a/tests/test_eager_first_boot_pgw671.py +++ b/tests/test_eager_first_boot_pgw671.py @@ -311,40 +311,79 @@ async def _run() -> None: asyncio.run(_run()) -def test_kill_switch_restores_the_sequential_gate_and_measures_the_split( +def test_eager_first_is_unconditional_and_no_env_restores_the_sequential_gate( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - """RED-VERIFICATION of the gate removal: with the env kill switch the - old ladder is back — every compile lands BEFORE READY — and the - elapsed-boot split is the eager-vs-compiled latency evidence.""" + """pgw#995 RED-VERIFICATION, inverted: the surviving path is the ONLY path. + + This test used to drive `GEN_WORKER_EAGER_FIRST_BOOT=0` to restore the + pre-pgw#671 sequential ladder, and the elapsed-boot split between the two + arms was the eager-vs-compiled latency evidence. That switch is deleted: + it defaulted ON, no release ever declared it and no endpoint ever set it, + so deleting it made the shape every pod already ran unconditional. + + An env used as a red-verification seam is the same defect as an env used as + a feature gate — it is a behaviour switch that a rebuild can flip, and + `GEN_WORKER_PREFER_AOT` is what that costs. So the assertion inverts: rather + than proving the OFF arm still works, prove the OFF arm is UNREACHABLE. + Setting the old name (or any plausible spelling of it) must change nothing. + + The latency evidence the old second arm carried is not lost — it is the + absolute bound below, which is the half that actually stated the claim: + time-to-READY does not pay the compile wall. The old test asserted BOTH + `ready_at < 3 * delay` and `ready_at < h_off.ready_at`; the second is + implied by the first whenever the sequential arm is honest, so the arm was + paying for a comparison the bound already made. + """ delay = 0.3 - monkeypatch.setenv("GEN_WORKER_EAGER_FIRST_BOOT", "0") - h_off = _Harness(tmp_path / "off", monkeypatch, compile_delay_s=delay) - - async def _off() -> None: - await h_off.boot() - assert h_off.rec.ready is True - assert h_off.rec.background_mint is None - # Sequential: the full plan compiled foreground, gating READY. - assert len(h_off.compile_log) == 3 - assert h_off.ex.serving_tiers() == {"generate": "compiled"} - - asyncio.run(_off()) - assert h_off.ready_at is not None and h_off.ready_at >= 3 * delay - - monkeypatch.setenv("GEN_WORKER_EAGER_FIRST_BOOT", "1") - h_on = _Harness(tmp_path / "on", monkeypatch, compile_delay_s=delay) - - async def _on() -> None: - await h_on.boot() - assert h_on.rec.background_mint is not None - await h_on.wait_mint() - - asyncio.run(_on()) - assert h_on.ready_at is not None - # The whole point: time-to-READY no longer pays the compile wall. - assert h_on.ready_at < 3 * delay - assert h_on.ready_at < h_off.ready_at + for name in ( + "GEN_WORKER_EAGER_FIRST_BOOT", + "GEN_WORKER_EAGER_FIRST", + "GEN_WORKER_EAGER_FIRST_BOOT_OFF", + ): + monkeypatch.setenv(name, "0") + + h = _Harness(tmp_path / "on", monkeypatch, compile_delay_s=delay) + + async def _run() -> None: + await h.boot() + # The pre-pgw#671 ladder would have compiled the full plan in the + # foreground and left no background mint at all. + assert h.rec.background_mint is not None, ( + "an env turned eager-first off — the pgw#995 deletion did not take, " + "or a new switch was added on top of it") + assert h.rec.ready is True + await h.wait_mint() + + asyncio.run(_run()) + assert h.ready_at is not None + # The whole point, stated as an absolute bound rather than a comparison + # against an arm that no longer exists: READY does not wait for the wall. + assert h.ready_at < 3 * delay + + +def test_no_env_read_survives_in_the_eager_first_and_bg_yield_paths() -> None: + """pgw#995 structural guard: the deleted names are gone from the SOURCE. + + A behavioural assertion can pass while a second, unreached reader survives + somewhere else in the module — which is exactly how `GEN_WORKER_PREFER_AOT` + kept two live gates after one was believed removed. So read the source. + """ + import gen_worker.executor as _ex + import gen_worker.mint_delegate as _md + import gen_worker.aot_wrapper_split as _ws + + src = "".join( + Path(m.__file__).read_text() for m in (_ex, _md, _ws) if m.__file__) + for gone in ( + "GEN_WORKER_EAGER_FIRST_BOOT", + "GEN_WORKER_BG_YIELD", + "GEN_WORKER_AOT_WRAPPER_SPLIT_OFF", + ): + # Prose that NAMES the deleted switch is fine and wanted; a read is not. + assert f'environ.get("{gone}"' not in src, ( + f"{gone} is read again — pgw#995 deleted it because env must carry " + f"config and secrets, never a branch selection") def test_mid_build_abandonment_is_clean_and_keeps_serving_eager( diff --git a/tests/test_hub_env_delivery_pgw995.py b/tests/test_hub_env_delivery_pgw995.py new file mode 100644 index 00000000..a68501c8 --- /dev/null +++ b/tests/test_hub_env_delivery_pgw995.py @@ -0,0 +1,241 @@ +"""pgw#995 deliverable 2 — hub-shaped env delivery reaches `Settings`. + +The `GEN_WORKER_PREFER_AOT` postmortem was not a bug in any component. The flag +was declared, the entry was set, the loader worked, the gate worked. What broke +was the DELIVERY between them: a release rebuild stopped declaring the name, the +hub withheld the entry silently, and the worker booted without it. Three pod +attempts. + +Nothing in this repo could have caught that, because the local rig constructs +its own environment — `mint_process.child_env` for the mint child, +`dict(os.environ)` for the adopting process. Both are shapes a production pod +never has. So the regression class had exactly one detector: a pod. + +These tests give it a second one. They drive the REAL +`gen_worker.config.load_settings` — the same function `entrypoint._run_main` +calls — over an environment produced by the hub's resolution rule rather than by +the test's own convenience. + +Run: pytest tests/test_hub_env_delivery_pgw995.py -v +""" + +from __future__ import annotations + +import pytest + +from gen_worker import config as config_pkg +from gen_worker.config import load_settings + +from harness import hub_env + + +def _boot(monkeypatch: pytest.MonkeyPatch, env: dict) -> None: + """Replace the process environment with a pod's, exactly.""" + for name in list(os_environ_names()): + monkeypatch.delenv(name, raising=False) + for k, v in env.items(): + monkeypatch.setenv(k, v) + + +def os_environ_names() -> list: + import os + + return [n for n in os.environ if n.startswith( + ("GEN_WORKER_", "TENSORHUB_", "WORKER_", "COZY_", "HF_"))] + + +# --------------------------------------------------------------------------- +# The seam: a DECLARED entry is delivered and reaches the typed struct +# --------------------------------------------------------------------------- + + +def test_a_hub_delivered_env_value_reaches_settings( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The whole chain, end to end, with nothing hand-placed in the middle. + + The worker function declares `HF_TOKEN`; the operator sets it; the release + declares it; so the hub delivers it and `Settings` carries it. `HF_TOKEN` is + the realistic subject rather than a made-up name: it is org-settable (the + hub reserves the `HF_` prefix but exempts this one), it is what a gated + clone needs, and its absence is what th#1073 records breaking gated clones + for days. + + Every hop is the production one — `resolve()` implements the hub's rule and + `load_settings()` is the function the worker's own entrypoint calls. + """ + declarations = hub_env.declared_by(["HF_TOKEN"]) + entries = hub_env.EndpointEnvEntries( + {"HF_TOKEN": "hf_operator_set_token"}) + + delivery = hub_env.resolve(declarations, entries) + assert delivery.env == {"HF_TOKEN": "hf_operator_set_token"} + assert delivery.withheld == () + + # The pod boots with image env + delivered entries, and NOTHING the test + # process happened to be carrying. + _boot(monkeypatch, hub_env.pod_environ({}, delivery)) + + settings = load_settings() + assert settings.hf_token == "hf_operator_set_token", ( + "a declared, delivered env did not reach Settings — the delivery chain " + "is broken between the hub's resolve and config.loader") + + +# --------------------------------------------------------------------------- +# The regression: the postmortem's exact shape, now RED locally +# --------------------------------------------------------------------------- + + +def test_a_rebuild_that_stops_declaring_a_name_withholds_it_and_says_so( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """`GEN_WORKER_PREFER_AOT`, reproduced in milliseconds instead of three pods. + + Release N declares the name and the value arrives. Release N+1 is rebuilt + from worker code that no longer declares it — the operator changed nothing, + the entry is untouched — and the value stops arriving. The point is not that + it stops (that is the declaration contract working); the point is that the + delivery SAYS SO, which is the half that was missing on both sides. + """ + entries = hub_env.EndpointEnvEntries( + {"HF_TOKEN": "hf_operator_set_token"}) + + before = hub_env.resolve( + hub_env.declared_by(["HF_TOKEN"]), entries) + _boot(monkeypatch, hub_env.pod_environ({}, before)) + assert load_settings().hf_token == "hf_operator_set_token" + + # The rebuild. Nobody edited the entry; the worker function's env list + # changed, so the release declares nothing. + after = hub_env.resolve(hub_env.declared_by([]), entries) + + assert after.env == {}, "an undeclared name must not be injected" + assert after.withheld_names() == ["HF_TOKEN"], ( + "the rebuild dropped a configured entry and reported NOTHING — this is " + "precisely the silence that cost three pod attempts (th#1650)") + assert after.withheld[0].reason == hub_env.WITHHELD_UNDECLARED + assert "0 env name(s)" in after.withheld[0].detail, ( + "a withholding must distinguish 'this release declares nothing at all' " + "(the rebuild case) from 'this one name was removed' (the intended one)") + + _boot(monkeypatch, hub_env.pod_environ({}, after)) + assert load_settings().hf_token == "", ( + "the withheld value still reached Settings — something other than the " + "hub is supplying it, which is the substitution this harness exists to " + "forbid") + + +def test_an_ambient_export_cannot_stand_in_for_a_hub_delivered_value( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A rig that lets the developer's own shell satisfy the assertion is + decoration. `pod_environ(strip=...)` is what stops that, so it is tested + rather than trusted: with the name exported ambiently and NOT declared by + the release, `Settings` must still come up empty. + """ + monkeypatch.setenv("HF_TOKEN", "ambient_shell_token") + ambient = {"HF_TOKEN": "ambient_shell_token"} + + delivery = hub_env.resolve( + hub_env.declared_by([]), + hub_env.EndpointEnvEntries({"HF_TOKEN": "hf_operator_set_token"})) + + env = hub_env.pod_environ(ambient, delivery, strip=["HF_TOKEN"]) + _boot(monkeypatch, env) + assert load_settings().hf_token == "" + + +# --------------------------------------------------------------------------- +# The reserved namespace still wins over a declaration +# --------------------------------------------------------------------------- + + +def test_a_release_cannot_declare_its_way_into_the_platform_namespace( +) -> None: + """pgw#763 delta 0: the process-split switch is platform-only. Declaring it + must not deliver it — otherwise a release could opt its own tenant code out + of the boundary that contains it.""" + delivery = hub_env.resolve( + hub_env.declared_by(["GEN_WORKER_COMPUTE_CHILD"]), + hub_env.EndpointEnvEntries({"GEN_WORKER_COMPUTE_CHILD": "1"})) + assert delivery.env == {} + assert delivery.withheld[0].reason == hub_env.WITHHELD_RESERVED + + +def test_the_loader_is_the_only_component_this_harness_talks_to() -> None: + """Guard against the harness growing into a second config implementation. + + `hub_env` models DELIVERY. The moment it starts constructing `Settings` + itself it stops testing the production path and starts certifying its own — + which is how a harness becomes the thing it was written to check. + """ + src = (hub_env.__file__ or "") + assert src + text = open(src).read() + for forbidden in ("load_settings", "Settings(", "msgspec"): + assert forbidden not in text, ( + f"hub_env references {forbidden!r}: it must produce an ENVIRONMENT " + f"and let the real loader turn it into config") + assert config_pkg.load_settings is load_settings + + +# --------------------------------------------------------------------------- +# The rig mode itself (pgw#995 deliverable 2) +# --------------------------------------------------------------------------- + + +def _rig(): + import sys + from pathlib import Path + + root = Path(__file__).resolve().parents[1] + sys.path.insert(0, str(root / "scripts")) + import micro_mint_rig # noqa: PLC0415 - the script under test + + return micro_mint_rig + + +def test_rig_hub_env_mode_delivers_declared_and_strips_ambient() -> None: + """`--hub-env` boots the mint child the way a pod is booted. + + Two properties, and the second is the one that makes the mode worth having: + a DECLARED entry is delivered, and an AMBIENT value of the same name is + stripped rather than inherited. Without the strip, a developer with the + variable exported in their shell gets a green rig for a release that would + have booted without it on a pod — which is not a weaker test, it is a test + that reports the opposite of the truth. + """ + rig = _rig() + env, withheld = rig.hub_delivered_env( + {"PATH": "/usr/bin", "HF_TOKEN": "ambient-shell-value"}, + {"HF_TOKEN": "delivered-by-hub"}) + + assert env["HF_TOKEN"] == "delivered-by-hub" + assert env["PATH"] == "/usr/bin", "image env must survive" + assert withheld == [] + + +def test_rig_hub_env_mode_reports_an_undeclared_entry_instead_of_dropping_it( +) -> None: + """The rig's whole reason to exist is turning a pod-only failure into a + local one. An entry the release does not declare must show up as a FACT the + rig reports, not as a variable that quietly is not there.""" + rig = _rig() + env, withheld = rig.hub_delivered_env( + {"PATH": "/usr/bin"}, {"COZY_SOMETHING_UNDECLARED": "x"}) + + assert "COZY_SOMETHING_UNDECLARED" not in env + assert len(withheld) == 1 + assert withheld[0]["name"] == "COZY_SOMETHING_UNDECLARED" + assert withheld[0]["reason"] == hub_env.WITHHELD_UNDECLARED + assert withheld[0]["detail"], "a withholding with no detail is a shrug" + + +def test_rig_strips_every_name_it_claims_to_deliver() -> None: + """A name the rig DECLARES but does not STRIP is a hole: ambient value in, + hub value never exercised, mode silently decorative.""" + rig = _rig() + assert set(rig.RIG_DECLARED_ENV) <= set(rig.RIG_STRIPPED_ENV), ( + "every declared name must also be stripped, or the ambient environment " + "can satisfy the assertion the mode exists to make") diff --git a/tests/test_mint_gate_pgw677.py b/tests/test_mint_gate_pgw677.py index 2509ff57..bfa9c92c 100644 --- a/tests/test_mint_gate_pgw677.py +++ b/tests/test_mint_gate_pgw677.py @@ -4,19 +4,23 @@ mint work yields: 1. STARVATION SHAPE (the live incident): during a background mint, - tenant requests complete at serving latency. RED-verified via the - GEN_WORKER_BG_YIELD=0 kill switch, which restores the pre-fix tree: - mint seed units inline-compile while holding the per-instance run - gate (the router's headroom degrade), so a tenant request queues for - the length of the unit — the measured "completions land exactly as - mint units free the gate / 0 renders in 19 min" shape. + tenant requests complete at serving latency. This used to be + RED-verified against the pre-fix tree, reachable via a + GEN_WORKER_BG_YIELD=0 kill switch: mint seed units inline-compiled + while holding the per-instance run gate (the router's headroom + degrade), so a tenant request queued for the length of the unit — + the measured "completions land exactly as mint units free the gate / + 0 renders in 19 min" shape. pgw#995 DELETED that switch and the tree + it selected, so the shape is now unreachable rather than merely + unselected, and the property is asserted in absolute terms against + the harness's own configured quantities. 2. RACE EXCLUSION (the pgw#676 SIGSEGV class): the shape-warm thread's compile can never execute the shared modules concurrently with a tenant forward. Post-fix the compile owns a background turn (single-flight, instance turn_mutex, tenant-quiet admission); the tenant that arrives mid-compile waits — bounded by ONE compile — and its wait is attributed to `instance_gate_wait`, never to runtime_ms. - RED-verified: with the kill switch the overlap is observed. + Structurally impossible post-pgw#995, not merely not-selected. 3. MINIMUM PROGRESS: under a sustained tenant stream the mint still finishes — the steal rule grants one bounded background unit per debt window; stolen units are not preemptible. @@ -328,38 +332,19 @@ def test_tenant_serves_at_serving_latency_during_mint_and_red_verifies( # headroom inside its exclusive turn instead. monkeypatch.setattr(hot_swap, "_headroom_ok", lambda device: False) - # --- RED: pre-fix tree via the kill switch --------------------------- - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "0") - h_off = _Harness( - tmp_path / "off", monkeypatch, - compile_delay_s=0.15, seed_forward_s=0.6, tenant_forward_s=0.02) + # pgw#995: the RED arm here drove `GEN_WORKER_BG_YIELD=0` to restore the + # pre-pgw#677 tree (idle-gate between units, bare run gate around the + # forward, inline compiles on handler workers) and asserted the starvation + # shape it produced. The switch is deleted — it defaulted ON, no release + # ever declared it and no endpoint ever set it — so the legacy tree it + # selected is deleted with it and there is nothing left to red-verify + # against. An env kept alive to be a test seam is still an env that selects + # behaviour, and a release rebuild cannot tell the two purposes apart. + # + # What survives is the arm that states the property in absolute terms, and + # `test_no_env_restores_the_pre_pgw677_tree` below proves the deleted arm is + # unreachable rather than merely unused. - async def _off() -> Tuple[float, pb.JobResult]: - await h_off.boot() - assert h_off.rec.background_mint is not None - # Wait until a BACKGROUND mint seed unit is in flight, holding the - # run gate (the boot's foreground eager pass contributed run #1). - deadline = time.monotonic() + 10.0 - while len(h_off.seed_runs) < 2 and time.monotonic() < deadline: - await asyncio.sleep(0.01) - assert len(h_off.seed_runs) >= 2, "mint never started seeding" - res, wall = await h_off.dispatch("r-red", aspect="16:9") - assert res.status == pb.JOB_STATUS_OK, res.safe_message - return wall, res - - wall_off, _res_off = asyncio.run(_off()) - # The collapse shape: the request queued behind the seed unit's - # inline compile (and then paid its own inline compile on the - # degraded router) — an order of magnitude over serving latency. - assert wall_off >= 0.5, ( - f"expected the pre-fix starvation shape, got {wall_off:.3f}s") - # Pre-fix, inline compiles ran on handler (to_thread) workers. - assert any( - "shape-warm" not in thread for _, thread, _, _ in h_off.compiles), ( - "pre-fix tree should compile inline off the warm thread") - - # --- POST-FIX --------------------------------------------------------- - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h_on = _Harness( tmp_path / "on", monkeypatch, compile_delay_s=0.15, seed_forward_s=0.6, tenant_forward_s=0.02) @@ -423,30 +408,14 @@ def test_compile_and_tenant_forward_never_overlap_and_red_verifies( segfaulted sm_86 (pgw#676: _forward_with_branch racing compile_wrapper) is structurally impossible — and the wait is attributed to the instance_gate_wait stage, not billed as runtime. RED: with the kill - switch the overlap is observed.""" - - # --- RED: pre-fix tree — overlap happens ----------------------------- - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "0") - h_off = _Harness( - tmp_path / "off", monkeypatch, - compile_delay_s=0.8, seed_forward_s=0.0, tenant_forward_s=0.05) - - async def _off() -> None: - await h_off.boot() - deadline = time.monotonic() + 10.0 - while not h_off.in_compile.is_set(): - assert time.monotonic() < deadline, "no background compile ran" - await asyncio.sleep(0.005) - res, _wall = await h_off.dispatch("r-race", aspect="1:1") - assert res.status == pb.JOB_STATUS_OK, res.safe_message + switch the overlap is observed (pgw#995: that arm is gone — see below).""" - asyncio.run(_off()) - assert h_off.overlaps, ( - "pre-fix tree must exhibit the pgw#676 overlap (tenant forward " - "concurrent with the warm thread's compile)") + # pgw#995: the RED arm drove `GEN_WORKER_BG_YIELD=0` and asserted that the + # pre-fix tree DOES exhibit the pgw#676 overlap. The switch and that tree + # are deleted, so the overlap is now structurally impossible rather than + # merely not-selected — which is the stronger statement the docstring + # already made. - # --- POST-FIX: exclusion holds, wait is attributed ------------------- - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h_on = _Harness( tmp_path / "on", monkeypatch, compile_delay_s=0.8, seed_forward_s=0.0, tenant_forward_s=0.05) @@ -492,7 +461,6 @@ def test_mint_completes_under_sustained_tenant_load( """A tenant stream that never drains would starve a purely idle-gated mint forever. The steal rule grants one bounded background unit per debt window: the mint completes while the stream keeps serving.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") monkeypatch.setattr(executor_mod, "_BG_STEAL_FLOOR_S", 0.1) monkeypatch.setattr(executor_mod, "_BG_STEAL_DEBT_FACTOR", 1.0) monkeypatch.setattr(executor_mod, "_BG_COMPILE_QUIESCENCE_S", 0.0) @@ -572,3 +540,25 @@ def compiled(*args: Any, **kwargs: Any) -> None: legacy.enable() verdict, _sig = legacy.route("t", compiled, ("d",), {}) assert verdict == hot_swap.COMPILED + + +# --------------------------------------------------------------------------- +# pgw#995 — the deleted arm is UNREACHABLE, not merely unused +# --------------------------------------------------------------------------- + + +def test_no_env_restores_the_pre_pgw677_tree() -> None: + """The two RED arms above were driven by an env. Deleting a switch and + deleting its tests looks identical to deleting a switch and leaving a + second reader behind — which is how `GEN_WORKER_PREFER_AOT` kept a live + gate after one was believed removed. So assert on the SOURCE. + """ + from pathlib import Path as _P + import gen_worker.executor as _ex + + src = _P(_ex.__file__).read_text() + assert 'environ.get("GEN_WORKER_BG_YIELD"' not in src, ( + "GEN_WORKER_BG_YIELD is read again — env carries config and secrets, " + "never a branch selection") + assert "_bg_yield_enabled" not in src, ( + "the bg-yield predicate is back; pgw#677's shape is unconditional") diff --git a/tests/test_mint_reopen_pgw677.py b/tests/test_mint_reopen_pgw677.py index f7bab584..5a60d3d3 100644 --- a/tests/test_mint_reopen_pgw677.py +++ b/tests/test_mint_reopen_pgw677.py @@ -333,7 +333,6 @@ def test_w8a8_stamp_with_hub_execution_lane_boots_eager_first( setup ran the FOREGROUND compile-then-serve mint (rec.background_mint is None), and the first tenant request sat behind the whole inline plan — the measured 26-minute cold-L4 starvation.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, compile_delay_s=0.4, seed_forward_s=0.05, tenant_forward_s=0.02, @@ -365,7 +364,6 @@ def test_true_mandatory_execution_lane_still_refuses_eager_first( """The qwen shape: hub lane says real w8a8 activations — eager is not a production tier there; the boot keeps the sequential foreground proof. Guards the fix from over-rotating.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, compile_delay_s=0.0, seed_forward_s=0.0, tenant_forward_s=0.01, @@ -384,7 +382,6 @@ def test_w8a8_stamp_without_execution_lane_evidence_stays_foreground( ) -> None: """No hub lane evidence: the weight-lane stamp remains the fail-closed fallback — unchanged pre-reopen behavior.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, weight_lane="w8a8", hub_execution_lane="") @@ -412,7 +409,6 @@ def test_multi_minute_compile_never_steals_against_live_demand( tenant completes at serving latency. RED on 0.70.0: the single 30s floor (patched to 0.1s, as tape 3 always did) lets the compile steal almost immediately and a tenant waits out the whole unabortable unit.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") monkeypatch.setattr(executor_mod, "_BG_STEAL_FLOOR_S", 0.1) # raising=False: on the pre-fix tree this attribute does not exist and # the single floor governs — that IS the red run. @@ -459,7 +455,6 @@ def test_compile_steal_is_announced_on_the_wire( """When the compile floor DOES elapse under truly continuous demand, the steal happens (minimum progress) and announces itself as a typed ``bg_turn_steal`` event.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") monkeypatch.setattr(executor_mod, "_BG_STEAL_FLOOR_S", 0.05) monkeypatch.setattr( executor_mod, "_BG_COMPILE_STEAL_FLOOR_S", 0.3, raising=False) @@ -503,7 +498,6 @@ def test_pack_or_closure_refusal_reaches_the_wire( (closure gate / pack) died in pod logs. Post-fix the verbatim reason rides the wire as a typed ``self_mint_abort`` event. RED on 0.70.0: no such event exists anywhere.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") def _refuse(*args: Any, **kwargs: Any) -> None: raise ValueError("closure refused: guard leak L['scale']") @@ -599,7 +593,6 @@ def test_oom_truncated_plan_never_finalizes_partial_capture( partial capture, and serving stays eager and alive. RED on 0.70.0: the OOM'd pass satisfied the stats-stable convergence and the mint finalized (phase=finalize) with nothing publishable and no event.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, compile_delay_s=0.02, seed_forward_s=0.01, tenant_forward_s=0.01, diff --git a/tests/test_mint_vram_budget_pgw737.py b/tests/test_mint_vram_budget_pgw737.py index 6fd972c3..6d3b9177 100644 --- a/tests/test_mint_vram_budget_pgw737.py +++ b/tests/test_mint_vram_budget_pgw737.py @@ -365,7 +365,6 @@ def test_mint_declines_on_a_card_that_cannot_hold_the_capture( tenant with it. Post-fix: no seed runs at all, one structured ``self_mint_skipped`` line on the wire, the tier stays eager, the cell stays absent, and the request SUCCEEDS.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness(tmp_path, monkeypatch, compile_delay_s=0.05) _card(monkeypatch, total_gib=79.19, resident_gib=54.2, peak_gib=65.4) @@ -404,7 +403,6 @@ def test_small_resident_still_mints_on_the_same_rig( ) -> None: """The fence: an sdxl-class residency has the headroom, so the budget must stay out of the way — the mint runs and the tier flips compiled.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness(tmp_path, monkeypatch, compile_delay_s=0.02) _card(monkeypatch, total_gib=23.6, resident_gib=7.0, peak_gib=9.5) @@ -432,7 +430,6 @@ def test_tenant_oom_evicts_the_mint_and_the_request_completes( five times and bought a second H100 for a deterministic failure. Post fix the MINT loses: it is abandoned, its targets unwrapped, the card freed, and the same request re-runs eager to OK on this same worker.""" - monkeypatch.setenv("GEN_WORKER_BG_YIELD", "1") h = _Harness( tmp_path, monkeypatch, compile_delay_s=1.5, seed_forward_s=0.05, tenant_oom_once=True) From e6804ae4da2457206c8cc4c5ba31eae0d2b35f27 Mon Sep 17 00:00:00 2001 From: Paul Fidika Date: Thu, 6 Aug 2026 23:57:59 -0600 Subject: [PATCH 2/2] pgw#995: the last reader of the deleted v1 kill switch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI caught what the local run had not reached: test_aot_wrapper_split_pgw793 referenced ws.DISABLE_ENV, the GEN_WORKER_AOT_WRAPPER_SPLIT_OFF constant deleted with the switch. The test asserted install() could be killed by the env; it now asserts the opposite — that the retired name is inert and the constant is gone. Which is the point rather than a formality: a test that merely stopped mentioning the switch would pass identically whether the deletion took or left a live reader behind somewhere else, and that ambiguity is exactly how GEN_WORKER_PREFER_AOT kept a live gate after one was believed removed. --- tests/test_aot_wrapper_split_pgw793.py | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/tests/test_aot_wrapper_split_pgw793.py b/tests/test_aot_wrapper_split_pgw793.py index e090ca1f..2249fc95 100644 --- a/tests/test_aot_wrapper_split_pgw793.py +++ b/tests/test_aot_wrapper_split_pgw793.py @@ -420,11 +420,20 @@ def digests() -> tuple[str, str]: ws._installed = False -def test_install_is_idempotent_and_killable(monkeypatch) -> None: # type: ignore[no-untyped-def] +def test_install_is_idempotent_and_no_env_can_kill_it(monkeypatch) -> None: # type: ignore[no-untyped-def] + """pgw#995: the `GEN_WORKER_AOT_WRAPPER_SPLIT_OFF` arm is DELETED. + + It defaulted ON, no release ever declared it and no endpoint ever set it, so + deleting it made the shape every pod already ran unconditional. Setting the + retired name must now be inert — asserted rather than assumed, because a + deletion that leaves a live reader elsewhere looks exactly like this test + passing for the wrong reason. + """ monkeypatch.setattr(ws, "_installed", False) - monkeypatch.setenv(ws.DISABLE_ENV, "1") - assert ws.install() is False - monkeypatch.delenv(ws.DISABLE_ENV) + monkeypatch.setenv("GEN_WORKER_AOT_WRAPPER_SPLIT_OFF", "1") + assert not hasattr(ws, "DISABLE_ENV"), ( + "the v1 kill switch is back; env carries config and secrets, never a " + "branch selection") torch_inductor = pytest.importorskip("torch._inductor.cpp_builder") original = torch_inductor.run_compile_cmd try: