feat(substrate): a queued execution past maxQueueAgeMs times out; ADRs 0004, 0010, 0012 match the code

apps/docs/reference/substrate-contract/README.md

@@ -29,7 +29,8 @@ Bring the keyed environment to the state the recipe describes.
 `{mode:'refuse'}` fails fast with `admission-refused` when the pool is
 full; `{mode:'queue'}` expects the consumer to have driven
 admissionEnqueue/Attempt to admission first, and refuses (never blocks)
-when it has not.
+when it has not. A queued execution that has waited `maxQueueAgeMs` or
+longer is refused with `timedOut: true` and loses its queue row.
 
 ###### Parameters
 
@@ -747,6 +748,7 @@ type AdmissionRefused = {
   position?: number;
   queuedForMs?: number;
   retryAfterMs?: number;
+  timedOut?: boolean;
 };
 ```
 
@@ -796,6 +798,16 @@ optional queuedForMs?: number;
 optional retryAfterMs?: number;
 ```
 
+##### timedOut?
+
+```ts
+optional timedOut?: boolean;
+```
+
+`true` when a `{mode:'queue'}` execution has waited `maxQueueAgeMs` or longer
+without admission. The substrate has released its queue row, so polling
+again re-enqueues at the back; stop and report instead.
+
 ***
 
 ### ApprovalRequired
@@ -1185,7 +1197,7 @@ type DetachedStatus =
 };
 ```
 
-What a detached process is doing. `unknown` is its own state rather than an
+What a detached process is doing. `gone` is its own state rather than an
 error: a container that slept, restarted or was checkpointed no longer has
 the process, and a consumer polling from a durable step needs to tell that
 apart from "still running" without catching a throw.

apps/substrate/specs/adr/0004-admission-enforced-by-ticket.md

@@ -19,16 +19,18 @@ actionable reason; a CI run must queue with visible progress, hibernating in its
 
 ## Decision
 
-- The substrate mints an **admission ticket** (HMAC over executionKey + pool, TTL) on admit; a container
-  class refuses to boot without a valid ticket. Enforcement is at the container, not in the caller.
+- The substrate mints an **admission ticket** (HMAC over consumer, key, pool and expiry; 10-minute
+  TTL, re-minted on heartbeat and never extended in place) on admit; a container class refuses to
+  boot without a valid ticket. Enforcement is at the container, not in the caller.
 - One pool per image class — lean, browser, agent, task — each with its own cap; deploy-time
   validation asserts the cap-sum stays within the account Containers ceiling. That cap partition,
   not FIFO fairness, is what prevents CI starving interactive tasks.
 - Consumers drive queue waits via `admission.enqueue/attempt/release`, hibernating in their own
   durable machinery; `ensure()` never blocks on a queue. Admission mode is consumer-chosen:
   `{mode:'refuse'}` or `{mode:'queue', maxQueueAgeMs}`. Refusal/timeout is a typed error carrying
-  `{pool, poolBusy, cap, position?, queuedForMs?, retryAfterMs?}`; `poolStatus()` exposes per-pool,
-  per-consumer occupancy.
+  `{pool, poolBusy, cap, position?, queuedForMs?, retryAfterMs?, timedOut?}`. A queued execution
+  that has waited `maxQueueAgeMs` or longer is refused with `timedOut: true` and loses its queue
+  row; `poolStatus()` exposes per-pool, per-consumer occupancy.
 - The admission D1 is bound to the substrate worker only. Consumer-side quotas (fractalbot's
   per-conversation/per-user caps) stay consumer-side, ahead of the physical gate.
 

apps/substrate/specs/adr/0010-named-image-classes-policy-selected.md

@@ -2,7 +2,7 @@
 
 - **Status:** Proposed
 - **Date:** 2026-08-06
-- **Implementation:** `shipped` — `src/admission/pools.ts:120` `poolPolicyView` / `:140` `selectPool`, four image classes in `apps/substrate/wrangler.jsonc`, recipe keys frozen by `SUBSTRATE_RECIPE_KEYS`. Tests: `admission/pools.test.ts`, `container-config.test.ts`.
+- **Implementation:** `partial` — `src/admission/pools.ts:120` `poolPolicyView` / `:140` `selectPool`, four image classes in `apps/substrate/wrangler.jsonc`, recipe keys frozen by `SUBSTRATE_RECIPE_KEYS`. Selection reads the consumer only: fractalbot gets `task`, every other consumer `lean`. The `browser` and `agent` classes are deployed and unselectable until the dispatcher's run catalog runs on the facade with a per-run class policy. Tests: `admission/pools.test.ts`, `container-config.test.ts`.
 
 ## Context
 

apps/substrate/specs/adr/0012-processes-that-outlive-the-exec-fence.md

@@ -41,9 +41,11 @@ This is a decision about *semantics*, and it makes the surface small:
 - `startDetached(key, input)` starts a process and returns a substrate-assigned id. It runs
   `ensure()` behind the ticket gate and crosses the ADR-0007 approval floor exactly as `exec` does —
   a floor command started detached must not be a way around the floor — and it applies **no grant**.
-- `detachedStatus(key, processId)` answers running / exited-with-code / unknown. There is no
-  `waitForExit` on the facade: a consumer polls from its own durable steps, the same shape admission
-  already uses, because a Worker call that blocks for twenty minutes is not a call.
+- `detachedStatus(key, processId)` answers running / exited-with-code / gone, where `gone` carries
+  the reason the substrate no longer has the process (an id this execution never started, a process
+  the container no longer tracks, or a container that cannot be reached). There is no `waitForExit`
+  on the facade: a consumer polls from its own durable steps, the same shape admission already uses,
+  because a Worker call that blocks for twenty minutes is not a call.
 - `stopDetached(key, processId)` kills it and forgets it. `abort` and `checkpoint` clear every
   record with the container.
 

apps/substrate/src/facade.ts

@@ -162,7 +162,12 @@ export abstract class SubstrateFacadeBase extends WorkerEntrypoint<Env> implemen
       if (!admitted.admitted) {
         // Fail-fast refusal in both modes — in queue mode the consumer keeps
         // driving attempts from its own durable steps; this call never waits.
-        if (admission.mode === "refuse") await this.store.release(id);
+        // A queued execution past its age ceiling loses its row: the refusal
+        // is final, so the row would otherwise hold a place nobody claims.
+        const queuedForMs = Math.max(0, Date.now() - admitted.enqueuedAt);
+        const timedOut =
+          admission.mode === "queue" && queuedForMs >= admission.maxQueueAgeMs;
+        if (admission.mode === "refuse" || timedOut) await this.store.release(id);
         return {
           ok: false,
           refusal: {
@@ -171,8 +176,8 @@ export abstract class SubstrateFacadeBase extends WorkerEntrypoint<Env> implemen
             poolBusy: admitted.poolBusy,
             cap: this.caps[pool],
             position: admitted.position,
-            queuedForMs: Math.max(0, Date.now() - admitted.enqueuedAt),
-            retryAfterMs: ADMISSION_RETRY_AFTER_MS,
+            queuedForMs,
+            ...(timedOut ? { timedOut: true } : { retryAfterMs: ADMISSION_RETRY_AFTER_MS }),
           },
         };
       }

apps/substrate/src/facade.workers.test.ts

@@ -113,6 +113,43 @@ describe("admission (ADR-0004) - the ceiling is the substrate's to own", () => {
     expect(outcome.refusal).toHaveProperty("retryAfterMs");
   });
 
+  it("keeps a queued execution's row while it is under its age ceiling", async () => {
+    await dispatcher().admissionAttempt(freshKey(), RECIPE);
+    await dispatcher().admissionAttempt(freshKey(), RECIPE);
+
+    const waiting = freshKey();
+    const outcome = await dispatcher().ensureSandbox(waiting, RECIPE, {
+      mode: "queue",
+      maxQueueAgeMs: 60 * 60_000,
+    });
+    expect(outcome).toMatchObject({ ok: false, refusal: { kind: "admission-refused" } });
+    if (outcome.ok) return;
+    expect(outcome.refusal).toHaveProperty("retryAfterMs");
+    expect(outcome.refusal).not.toHaveProperty("timedOut");
+    const behind = await dispatcher().admissionEnqueue(freshKey(), RECIPE);
+    expect(behind.position).toBeGreaterThanOrEqual(1);
+  });
+
+  it("times a queued execution out past maxQueueAgeMs and releases its row", async () => {
+    await dispatcher().admissionAttempt(freshKey(), RECIPE);
+    await dispatcher().admissionAttempt(freshKey(), RECIPE);
+
+    const before = await dispatcher().poolStatus();
+    const queuedBefore = before.pools.find((p) => p.pool === "lean")?.queued ?? 0;
+    const outcome = await dispatcher().ensureSandbox(freshKey(), RECIPE, {
+      mode: "queue",
+      maxQueueAgeMs: 0,
+    });
+    expect(outcome).toMatchObject({
+      ok: false,
+      refusal: { kind: "admission-refused", pool: "lean", timedOut: true },
+    });
+    if (outcome.ok) return;
+    expect(outcome.refusal).not.toHaveProperty("retryAfterMs");
+    const after = await dispatcher().poolStatus();
+    expect(after.pools.find((p) => p.pool === "lean")?.queued).toBe(queuedBefore);
+  });
+
   it("partitions the pools by consumer policy, never by anything a caller says", async () => {
     // fractalbot lands on `task` (cap 1) and the dispatcher on `lean` (cap 2)
     // from the SAME recipe: the image class is policy-selected from the

packages/runtime-cf/src/sandbox-facade.ts

@@ -114,7 +114,9 @@ export const describeRefusal: (refusal: SubstrateRefusal) => string =
   Match.type<SubstrateRefusal>().pipe(
     Match.discriminatorsExhaustive("kind")({
       "admission-refused": (r) =>
-        `substrate admission refused: pool ${r.pool} is ${r.poolBusy}/${r.cap} busy`,
+        r.timedOut
+          ? `substrate admission timed out: queued ${r.queuedForMs ?? 0}ms behind pool ${r.pool} at ${r.poolBusy}/${r.cap} busy`
+          : `substrate admission refused: pool ${r.pool} is ${r.poolBusy}/${r.cap} busy`,
       "approval-required": (r) =>
         `substrate refused an irreversible command: the run definition does not pre-assert "${r.rule}"`,
       "attestation-rejected": (r) => `substrate rejected the approval attestation: ${r.reason}`,

packages/substrate-contract/src/index.ts

@@ -261,6 +261,12 @@ export type AdmissionRefused = {
   position?: number;
   queuedForMs?: number;
   retryAfterMs?: number;
+  /**
+   * `true` when a `{mode:'queue'}` execution has waited `maxQueueAgeMs` or longer
+   * without admission. The substrate has released its queue row, so polling
+   * again re-enqueues at the back; stop and report instead.
+   */
+  timedOut?: boolean;
 };
 
 /** The command matches the irreversible floor and no attestation was carried. */
@@ -373,7 +379,7 @@ export type DetachedProcess = {
 };
 
 /**
- * What a detached process is doing. `unknown` is its own state rather than an
+ * What a detached process is doing. `gone` is its own state rather than an
  * error: a container that slept, restarted or was checkpointed no longer has
  * the process, and a consumer polling from a durable step needs to tell that
  * apart from "still running" without catching a throw.
@@ -491,7 +497,8 @@ export interface SubstrateFacade {
    * `{mode:'refuse'}` fails fast with `admission-refused` when the pool is
    * full; `{mode:'queue'}` expects the consumer to have driven
    * admissionEnqueue/Attempt to admission first, and refuses (never blocks)
-   * when it has not.
+   * when it has not. A queued execution that has waited `maxQueueAgeMs` or
+   * longer is refused with `timedOut: true` and loses its queue row.
    */
   ensureSandbox(
     key: SandboxKey,