pgw#997: a REAL micro endpoint family — 3 export entries, a 15-second full mint cycle

Taskfile.yml

@@ -34,6 +34,19 @@ tasks:
     cmds:
       - nice -n 10 uv run python scripts/micro_mint_rig.py {{.CLI_ARGS}}
 
+  rig:micro:
+    desc: |
+      pgw#997 — the same full cycle against `examples/micro-diffusion`, the
+      org-worker-shaped endpoint: 3 export entries (two fork arms + a second
+      target), CONTAINER inputs with a plain input after them so every cycle
+      re-proves pgw#993/pgw#994, deterministically GENERATED weights, and a
+      parity check that arms the adopted cell and compares every arm to eager.
+
+      This is the vehicle a mint-path change should be proven against;
+      `task rig:mint` is the cheaper one-entry plumbing toy.
+    cmds:
+      - nice -n 10 uv run python scripts/micro_mint_rig.py --vehicle micro {{.CLI_ARGS}}
+
   rig:test:
     desc: |
       pgw#978 — the rig as a pytest row, including the full cycle (opt-in via

changelog.d/pgw997.md

@@ -0,0 +1,23 @@
+### Added
+
+- **A micro endpoint family, so an AOT mint cycle costs seconds instead of
+  hours (pgw#997).** `examples/micro-diffusion` is a real org worker — its own
+  `pyproject.toml`, `endpoint.toml`, Dockerfile-first build, catalog slot,
+  registered export declaration — that declares **3 export entries** over a
+  1.1 MB deterministically generated checkpoint. sdxl declares 36 and costs
+  ~95 minutes a pod cycle; the full local machinery cycle against this family
+  is **15 s**, including a parity check that arms the adopted cell in a second
+  process and compares every arm to eager (max |delta| 7.2e-07).
+- The declaration deliberately keeps the container-flattening seam under test
+  on every cycle: a `repeat=` container input with a **plain input immediately
+  after it** (pgw#994's shape), and one `Dim` carried by both a container
+  element axis and a plain tensor axis (pgw#993's).
+- **`task rig:micro`** runs the pgw#978 rig against it; `scripts/micro_mint_rig.py`
+  gained `--vehicle {tiny,micro}` and its endpoint coupling moved into
+  `tests/harness/rig_vehicles.py`. `task rig:mint` is unchanged.
+
+### Fixed
+
+- The rig's adopt leg now installs the same runtime probes the mint child got
+  and configures logging, so a filter miss reports the axis it was rejected on
+  instead of an empty log.

docs/probe-worker.md

@@ -11,7 +11,8 @@ Tier by tier, cheapest first:
 
 | Tier | Cost per iteration | Covers |
 |---|---|---|
-| `task rig:mint` (pgw#978) | seconds, free | resolve, handoff, spawn, load, warm, export, compile, seal, publish, adopt — on a toy model |
+| `task rig:mint` (pgw#978) | seconds, free | resolve, handoff, spawn, load, warm, export, compile, seal, publish, adopt — on a toy model, ONE export entry |
+| `task rig:micro` (pgw#997) | ~15 s, free | all of the above on a REAL org-worker package — 3 entries, two fork arms, a second target, container inputs, a derived dynamic range, plus a PARITY check that arms the adopted cell and compares every arm to eager |
 | experimental image (pgw#979) | one image build | the real image, the real deps, a real pod, a real family |
 | **probe pod (this doc)** | seconds, on a pod you already hold | everything above, on real weights and a real card, iterating |
 

examples/micro-diffusion/.dockerignore

@@ -0,0 +1,5 @@
+.venv/
+.mypy_cache/
+__pycache__/
+*.pyc
+.micro-weights/

examples/micro-diffusion/Dockerfile

@@ -0,0 +1,46 @@
+# The three Dockerfile contract points (docs/dockerfile.md), plus one thing
+# only this family does: it GENERATES its checkpoint at build time.
+#
+# The generation step is what keeps the family compliant with the
+# weights-locality rule while still being a real endpoint. There is no
+# checkpoint in git, none on a developer's box, and no download at boot — the
+# bytes are a pure function of a seed, materialized into the image once, and
+# byte-identical in every image built from this Dockerfile.
+ARG BASE_IMAGE=pytorch/pytorch:2.13.0-cuda13.0-cudnn9-runtime
+FROM ${BASE_IMAGE}
+
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
+
+WORKDIR /app
+
+COPY pyproject.toml uv.lock /app/
+
+RUN --mount=type=cache,id=cozy-uv-cache,target=/var/cache/uv,sharing=locked \
+    uv export --cache-dir /var/cache/uv --link-mode copy \
+      --no-dev --no-hashes --no-sources --no-emit-project --no-emit-local \
+      -o /tmp/requirements.txt \
+    && uv pip install --cache-dir /var/cache/uv --link-mode copy \
+      --system --break-system-packages --no-deps -r /tmp/requirements.txt
+
+COPY . /app
+
+RUN --mount=type=cache,id=cozy-uv-cache,target=/var/cache/uv,sharing=locked \
+    uv pip install --cache-dir /var/cache/uv --link-mode copy \
+      --system --break-system-packages --no-deps --no-sources /app
+
+# The checkpoint, generated and VERIFIED in the same layer. `--verify`
+# regenerates every tensor and byte-compares: an image whose weights did not
+# reproduce from the seed fails the build instead of shipping a checkpoint
+# nobody else can rebuild.
+ARG MICRO_SEED=997
+RUN python -m micro_diffusion.weights --out /app/.micro-weights \
+      --seed ${MICRO_SEED} --verify
+
+# Contract point 2: discovery baked into the image.
+ARG BUILD_NONCE=2026-08-06-default
+RUN echo "build-nonce=${BUILD_NONCE}" \
+    && mkdir -p /app/.tensorhub \
+    && python -m gen_worker.discovery > /app/.tensorhub/endpoint.lock
+
+# Contract point 3.
+ENTRYPOINT ["python", "-m", "gen_worker.entrypoint"]

examples/micro-diffusion/README.md

@@ -0,0 +1,198 @@
+# micro-diffusion — the fleet's smallest REAL endpoint family
+
+**Why it exists.** AOT-mint iteration has been using sdxl as its test vehicle:
+6.9 GB of weights, **36 export entries**, ~95 minutes per pod cycle. Nothing
+about the mint MACHINERY needs any of that. This family runs the identical
+machinery over a 1.1 MB generated checkpoint with **3 export entries**, so a
+change to the mint path can be proven in minutes instead of hours.
+
+It is a real org worker, not a fixture: its own `pyproject.toml`,
+`endpoint.toml`, Dockerfile-first build contract, catalog slot, registered
+export declaration, and two served arms. The only thing about it that is small
+is the model.
+
+```
+src/micro_diffusion/
+  model.py            MicroDenoiser (tiny DiT) + MicroDecoder — conv-free, register_buffer table
+  weights.py          deterministic seed -> checkpoint; `python -m micro_diffusion.weights`
+  pipeline.py         MicroPipeline: .denoiser / .decoder, from_pretrained
+  aot_declaration.py  the Compile declaration — 3 entries, container inputs
+  main.py             @endpoint Generate: generate (cfg on) / generate_turbo (cfg off)
+```
+
+## The three entries, and why exactly three
+
+```
+denoiser/cfg=true     the guided arm; container arity 2
+denoiser/cfg=false    the turbo arm; container arity 1
+decoder               a SECOND target, its own dims, no fork
+```
+
+That set is the smallest one that still exercises every decision the mint path
+makes: plan selection, a fork coordinate, a **derived dynamic range** (two
+latent rows collapse into one artifact per arm), a second target, entry naming,
+the seal, the publish wire, and the cross-process adopt filter.
+
+Two declared facts are there specifically to keep a known defect class under
+test on every cycle:
+
+* **A container input with a plain input immediately after it.** `x` is a
+  python `list[Tensor]` of arity N, `t` is a plain `(N,)` tensor, `cond` is a
+  second list. When `x` expands to N leaves, every contract position after it
+  shifts — which is exactly the divergence pgw#994 fixed (`input_contract`
+  records the FLATTENED position; `bind_call_inputs` was matching it against
+  the caller's PRE-flattening args). Any regression binds the whole list to
+  element 0 and the parity leg goes red in seconds.
+* **One dim carried by both a container element axis and a plain tensor axis.**
+  `H_lat` is `("x", 1)` for the denoiser and `("latent", 2)` for the decoder.
+  That is pgw#993's seam: `carried_by` has to resolve through the same
+  expansion `dynamic_shapes` mirrors.
+
+## Weights: generated, never fetched
+
+There is no checkpoint in git, none on a developer's box, and no download at
+boot. `micro_diffusion.weights` maps `(seed, config) -> bytes` deterministically
+and materializes the tree wherever it is needed — into the image at
+`docker build` time, on the pod at boot if the binding resolved to an empty
+tree, or into the local rig's scratch root.
+
+```bash
+python -m micro_diffusion.weights --out /tmp/w --verify   # 1.1 MB, reproduces from seed 997
+```
+
+`--verify` regenerates every tensor and byte-compares. The Dockerfile runs it
+with `--verify`, so an image whose weights did not reproduce fails the build
+rather than shipping a checkpoint nobody else can rebuild.
+
+## Local: the full cycle, in the rig
+
+The pgw#978 micro-mint rig drives the whole production mint path against this
+package on this box — no pod, no RunPod, no hub.
+
+```bash
+task rig:micro                      # the full cycle against micro-diffusion
+task rig:mint -- --vehicle micro    # same thing, long form
+task rig:mint                       # pgw#978's original one-entry plumbing toy
+```
+
+Legs: gates → weights → handoff → mint-child (a REAL spawned interpreter doing
+`torch.export` + AOTInductor) → publish (the real `CellPublisher` wire) →
+adopt+parity (a SECOND OS process discovering the cell, arming it, and
+comparing every arm against eager).
+
+---
+
+# ⛔ POD RUNBOOK — **DO NOT RUN WITHOUT PAUL'S EXPLICIT GO**
+
+Nothing below has been executed. It is written so that when the go comes, the
+run is a transcription rather than a design session. Every step is the same
+step attempt 26's sdxl runbook uses; only the sizes differ. **Timings marked
+ESTIMATE are derived, not measured** — measuring them is the first cycle's job.
+
+Preconditions (all verified before step 1, none assumed):
+
+- `task inflight STACK=dev` clean, 0 pods live, `max_concurrent_boots` is 1
+  fleet-wide so one `booting` row anywhere blocks this.
+- The hub's `platform_discretionary_budget` has headroom (a cell mint is
+  platform-paid work).
+- The wheel is a version that has completed a local `task rig:micro` cycle.
+  A wheel nothing has proven locally is what this whole family exists to stop.
+
+### 1. Wheel — pin the endpoint at the version under test
+
+`gen-worker==<version>` in `pyproject.toml`, `uv lock`, tar the directory.
+`endpoint.toml` is KEPT (it carries the real build profile).
+**ESTIMATE: < 1 min.** No commit to `inference-endpoints` is wanted — this is a
+proof artifact, not a fleet pin.
+
+> ⚠️ **The pin must carry pgw#994 (`2165c2d5`) — 0.93.4 or a master build.**
+> This family declares a container input with a plain input after it, which is
+> exactly the shape whose contract positions pgw#994 fixed. On 0.93.3 the cell
+> mints and seals and then **refuses at ingress on its first served call**, so
+> the run would burn a pod to rediscover a fixed defect. The local rig runs
+> against master and its parity leg is a green proof of that fix.
+
+### 2. Create the endpoint (first time only)
+
+```
+POST /api/v1/endpoints  {"owner":"tensorhub","name":"micro-diffusion"}
+```
+
+Auth: `POST /api/v1/password/login` → **`access_token`** (not `access`),
+15-minute expiry — refresh it around the build.
+
+### 3. Publish the release
+
+```
+POST /api/v1/endpoints/tensorhub/micro-diffusion/releases?dev=true&skip_profiling=true
+Content-Type: application/gzip     (raw tarball body)
+```
+
+202 → `{build_id, proposed_release_id}`; `200 {"status":"noop"}` is also
+success. Poll `GET /api/v1/endpoint-builds/{id}` to `succeeded`.
+**ESTIMATE: 2-3 min** (sdxl's measured build is 4m30s; this image installs no
+diffusers/transformers/accelerate and its weight-generation layer is ~1 s, so
+the delta is the dependency install).
+
+### 4. Tag `prod` — BEFORE the buy
+
+```
+PUT /api/v1/endpoints/tensorhub/micro-diffusion/tags/prod  {"release_id": …}
+```
+
+`compile-cells` reads the orchestrator's in-memory `LoadRelease` cache and
+answers `409 no_compile_declaration` for a release the runtime has not loaded.
+That 409 has twice been misread as a missing declaration; it is a cold cache.
+**ESTIMATE: seconds.**
+
+### 5. Arm mint boots
+
+`TENSORHUB_COMPILE_OBLIGATION_MINT_BOOTS=true`, restart the hub **process
+alone**. Confirm from the LOG, not the API:
+`[cell-obligation] loop started mint_boots=true`. The `"armed"` field in the
+`compile-cells` 202 reads current config, not the running loop's snapshot, and
+will lie to you. **ESTIMATE: < 1 min.**
+
+### 6. Buy
+
+```
+POST /v1/admin/compile-cells  {"release_id":…, "gpu_model":"L40S", "coverage":"warmup"}
+```
+
+Required, not optional: micro-diffusion is not a flagship family, so a
+publish-seeded obligation is `tier=lazy` and is never bought. The same call
+un-parks (its `ON CONFLICT (release_id, sku) DO UPDATE` resets `status`,
+`attempts`, `discharged_at`, `not_before`). **Buy it as FORGE** — 12 h worker
+JWT instead of 30 min, plus the activity-backstop and idle-turnover exemptions.
+
+### 7. Watch
+
+```
+GET /v1/admin/mints?release=…
+GET /v1/admin/worker-activity-events?kind=aot_mint_phases&release=…
+GET /v1/admin/fleet-status | .compile_cells
+```
+
+**ESTIMATE: 3-5 min pod-side**, decomposed rather than guessed:
+
+| phase | sdxl (measured) | micro (ESTIMATE) | why |
+|---|---|---|---|
+| pod boot + image pull | ~2 min | ~1-2 min | much smaller image |
+| weights download | minutes (6.9 GB) | **0 s** | generated into the image |
+| load + warm | ~1 min | seconds | 1.1 MB, 2 steps |
+| export + AOTI compile | ~90 min / 36 entries | ~2-3 min / 3 entries | 12x fewer entries, tiny graphs |
+| seal + publish | ~1 min | seconds | the cell is small |
+
+### 8. Adopt
+
+Drive demand so a SECOND pod boots cold and adopts the published cell from the
+hub. Bank the eager arm on a pod that is NOT concurrently minting.
+
+### 9. Safety half
+
+The mint is sm_89 (L40S). Boot an `a100-sxm4-80gb` (sm_80) and record the TYPED
+refusal — an untested refusal is not a guarantee.
+
+**Whole-cycle ESTIMATE: 5-8 minutes**, against sdxl's measured ~95. That
+number is the deliverable; the first real run replaces every ESTIMATE above
+with a measurement.

examples/micro-diffusion/endpoint.toml

@@ -0,0 +1,16 @@
+schema_version = 1
+name = "micro-diffusion"
+main = "micro_diffusion.main"
+
+[runtime]
+language = "python"
+
+# One profile, matching the fleet's current cuda/python/torch tuple, so a cell
+# minted here is adoptable by the same pool every other family serves from.
+# The image is small because the package pulls no diffusers/transformers.
+[[build.profiles]]
+name = "default"
+accelerator = "cuda"
+cuda = "13.0"
+python = "3.12"
+torch = "2.13.0"

examples/micro-diffusion/pyproject.toml

@@ -0,0 +1,51 @@
+[project]
+name = "micro-diffusion"
+version = "0.1.0"
+description = "The fleet's smallest REAL endpoint family — three export entries over a deterministically generated toy checkpoint, so a full AOT mint cycle costs minutes"
+requires-python = ">=3.12,<3.13"
+dependencies = [
+    # Hard-pinned per th#1098 fleet policy: compile cells verify() EXACT on
+    # the gen-worker version, so a range would make an adopted cell's identity
+    # depend on what the build happened to resolve.
+    #
+    # ⚠️ THIS FAMILY'S SERVE PATH REQUIRES pgw#994. It declares a `repeat=`
+    # container input with a plain input after it, which is precisely the
+    # shape whose contract positions pgw#994 fixed. On 0.93.3 the cell MINTS
+    # and SEALS and then refuses at ingress on its first served call — the
+    # same failure z-image's tracker entry describes. Bump this to the first
+    # release carrying `2165c2d5` (0.93.4, or a master build) before any pod
+    # run; the local rig already runs against master and its parity leg is
+    # currently a green proof of that fix.
+    "gen-worker==0.93.3",
+    "msgspec>=0.18",
+    "safetensors>=0.4",
+    "pillow",
+]
+
+# NO diffusers, NO transformers, NO accelerate. The whole point of this family
+# is that its image is small and its boot is short; a dependency it does not
+# use would cost both on every cycle.
+
+[dependency-groups]
+local = [
+    "torch>=2.11.0",
+]
+dev = [
+    "mypy>=1.10.0",
+    "pytest>=9.0.3",
+]
+
+[tool.gen_worker]
+main = "micro_diffusion.main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/micro_diffusion"]
+
+[tool.mypy]
+python_version = "3.12"
+ignore_missing_imports = true
+disallow_untyped_defs = true

examples/micro-diffusion/src/micro_diffusion/__init__.py

@@ -0,0 +1,22 @@
+"""micro-diffusion — the fleet's smallest REAL endpoint family.
+
+Its reason to exist is cycle time: a full production-path AOT mint against
+sdxl is 36 export entries and ~95 minutes on a pod, which makes every mint
+change a multi-hour experiment. This family declares THREE entries over a
+generated toy checkpoint and runs the identical machinery.
+"""
+
+from .aot_declaration import DECLARATION, FAMILY
+from .main import Generate, MicroDefaults, MicroIn, MicroOut, Size
+from .pipeline import MicroPipeline
+
+__all__ = [
+    "DECLARATION",
+    "FAMILY",
+    "Generate",
+    "MicroDefaults",
+    "MicroIn",
+    "MicroOut",
+    "MicroPipeline",
+    "Size",
+]