From 1527bddd16139788cf65660bae3054ddb4faf274 Mon Sep 17 00:00:00 2001 From: debuggingfuture Date: Thu, 24 Sep 2026 16:30:26 +0800 Subject: [PATCH] feat(substrate): a queued execution past maxQueueAgeMs times out and loses its row ensureSandbox in {mode:'queue'} refuses an execution that has waited maxQueueAgeMs or longer with `timedOut: true` and releases its queue row; under the ceiling the row stays and the refusal carries retryAfterMs as before. `timedOut` is an optional field on an output type, so CONTRACT_VERSION holds. ADR text matches the code: ADR-0004 names the ticket's signed fields (consumer, key, pool, expiry) and the queue timeout, ADR-0010 is `partial` with consumer-only pool selection, and ADR-0012 and the contract comment name the detached status `gone`. --- .../reference/substrate-contract/README.md | 16 +++++++- .../adr/0004-admission-enforced-by-ticket.md | 10 +++-- ...010-named-image-classes-policy-selected.md | 2 +- ...2-processes-that-outlive-the-exec-fence.md | 8 ++-- apps/substrate/src/facade.ts | 11 ++++-- apps/substrate/src/facade.workers.test.ts | 37 +++++++++++++++++++ packages/runtime-cf/src/sandbox-facade.ts | 4 +- packages/substrate-contract/src/index.ts | 11 +++++- 8 files changed, 83 insertions(+), 16 deletions(-) diff --git a/apps/docs/reference/substrate-contract/README.md b/apps/docs/reference/substrate-contract/README.md index 9f38dbb..ead71a3 100644 --- a/apps/docs/reference/substrate-contract/README.md +++ b/apps/docs/reference/substrate-contract/README.md @@ -29,7 +29,8 @@ Bring the keyed environment to the state the recipe describes. `{mode:'refuse'}` fails fast with `admission-refused` when the pool is full; `{mode:'queue'}` expects the consumer to have driven admissionEnqueue/Attempt to admission first, and refuses (never blocks) -when it has not. +when it has not. A queued execution that has waited `maxQueueAgeMs` or +longer is refused with `timedOut: true` and loses its queue row. ###### Parameters @@ -747,6 +748,7 @@ type AdmissionRefused = { position?: number; queuedForMs?: number; retryAfterMs?: number; + timedOut?: boolean; }; ``` @@ -796,6 +798,16 @@ optional queuedForMs?: number; optional retryAfterMs?: number; ``` +##### timedOut? + +```ts +optional timedOut?: boolean; +``` + +`true` when a `{mode:'queue'}` execution has waited `maxQueueAgeMs` or longer +without admission. The substrate has released its queue row, so polling +again re-enqueues at the back; stop and report instead. + *** ### ApprovalRequired @@ -1185,7 +1197,7 @@ type DetachedStatus = }; ``` -What a detached process is doing. `unknown` is its own state rather than an +What a detached process is doing. `gone` is its own state rather than an error: a container that slept, restarted or was checkpointed no longer has the process, and a consumer polling from a durable step needs to tell that apart from "still running" without catching a throw. diff --git a/apps/substrate/specs/adr/0004-admission-enforced-by-ticket.md b/apps/substrate/specs/adr/0004-admission-enforced-by-ticket.md index 3677326..e1126ac 100644 --- a/apps/substrate/specs/adr/0004-admission-enforced-by-ticket.md +++ b/apps/substrate/specs/adr/0004-admission-enforced-by-ticket.md @@ -19,16 +19,18 @@ actionable reason; a CI run must queue with visible progress, hibernating in its ## Decision -- The substrate mints an **admission ticket** (HMAC over executionKey + pool, TTL) on admit; a container - class refuses to boot without a valid ticket. Enforcement is at the container, not in the caller. +- The substrate mints an **admission ticket** (HMAC over consumer, key, pool and expiry; 10-minute + TTL, re-minted on heartbeat and never extended in place) on admit; a container class refuses to + boot without a valid ticket. Enforcement is at the container, not in the caller. - One pool per image class — lean, browser, agent, task — each with its own cap; deploy-time validation asserts the cap-sum stays within the account Containers ceiling. That cap partition, not FIFO fairness, is what prevents CI starving interactive tasks. - Consumers drive queue waits via `admission.enqueue/attempt/release`, hibernating in their own durable machinery; `ensure()` never blocks on a queue. Admission mode is consumer-chosen: `{mode:'refuse'}` or `{mode:'queue', maxQueueAgeMs}`. Refusal/timeout is a typed error carrying - `{pool, poolBusy, cap, position?, queuedForMs?, retryAfterMs?}`; `poolStatus()` exposes per-pool, - per-consumer occupancy. + `{pool, poolBusy, cap, position?, queuedForMs?, retryAfterMs?, timedOut?}`. A queued execution + that has waited `maxQueueAgeMs` or longer is refused with `timedOut: true` and loses its queue + row; `poolStatus()` exposes per-pool, per-consumer occupancy. - The admission D1 is bound to the substrate worker only. Consumer-side quotas (fractalbot's per-conversation/per-user caps) stay consumer-side, ahead of the physical gate. diff --git a/apps/substrate/specs/adr/0010-named-image-classes-policy-selected.md b/apps/substrate/specs/adr/0010-named-image-classes-policy-selected.md index 8d18ffd..ab68a1a 100644 --- a/apps/substrate/specs/adr/0010-named-image-classes-policy-selected.md +++ b/apps/substrate/specs/adr/0010-named-image-classes-policy-selected.md @@ -2,7 +2,7 @@ - **Status:** Proposed - **Date:** 2026-08-06 -- **Implementation:** `shipped` — `src/admission/pools.ts:120` `poolPolicyView` / `:140` `selectPool`, four image classes in `apps/substrate/wrangler.jsonc`, recipe keys frozen by `SUBSTRATE_RECIPE_KEYS`. Tests: `admission/pools.test.ts`, `container-config.test.ts`. +- **Implementation:** `partial` — `src/admission/pools.ts:120` `poolPolicyView` / `:140` `selectPool`, four image classes in `apps/substrate/wrangler.jsonc`, recipe keys frozen by `SUBSTRATE_RECIPE_KEYS`. Selection reads the consumer only: fractalbot gets `task`, every other consumer `lean`. The `browser` and `agent` classes are deployed and unselectable until the dispatcher's run catalog runs on the facade with a per-run class policy. Tests: `admission/pools.test.ts`, `container-config.test.ts`. ## Context diff --git a/apps/substrate/specs/adr/0012-processes-that-outlive-the-exec-fence.md b/apps/substrate/specs/adr/0012-processes-that-outlive-the-exec-fence.md index 4927844..3dca67e 100644 --- a/apps/substrate/specs/adr/0012-processes-that-outlive-the-exec-fence.md +++ b/apps/substrate/specs/adr/0012-processes-that-outlive-the-exec-fence.md @@ -41,9 +41,11 @@ This is a decision about *semantics*, and it makes the surface small: - `startDetached(key, input)` starts a process and returns a substrate-assigned id. It runs `ensure()` behind the ticket gate and crosses the ADR-0007 approval floor exactly as `exec` does — a floor command started detached must not be a way around the floor — and it applies **no grant**. -- `detachedStatus(key, processId)` answers running / exited-with-code / unknown. There is no - `waitForExit` on the facade: a consumer polls from its own durable steps, the same shape admission - already uses, because a Worker call that blocks for twenty minutes is not a call. +- `detachedStatus(key, processId)` answers running / exited-with-code / gone, where `gone` carries + the reason the substrate no longer has the process (an id this execution never started, a process + the container no longer tracks, or a container that cannot be reached). There is no `waitForExit` + on the facade: a consumer polls from its own durable steps, the same shape admission already uses, + because a Worker call that blocks for twenty minutes is not a call. - `stopDetached(key, processId)` kills it and forgets it. `abort` and `checkpoint` clear every record with the container. diff --git a/apps/substrate/src/facade.ts b/apps/substrate/src/facade.ts index d67953b..428aa04 100644 --- a/apps/substrate/src/facade.ts +++ b/apps/substrate/src/facade.ts @@ -162,7 +162,12 @@ export abstract class SubstrateFacadeBase extends WorkerEntrypoint implemen if (!admitted.admitted) { // Fail-fast refusal in both modes — in queue mode the consumer keeps // driving attempts from its own durable steps; this call never waits. - if (admission.mode === "refuse") await this.store.release(id); + // A queued execution past its age ceiling loses its row: the refusal + // is final, so the row would otherwise hold a place nobody claims. + const queuedForMs = Math.max(0, Date.now() - admitted.enqueuedAt); + const timedOut = + admission.mode === "queue" && queuedForMs >= admission.maxQueueAgeMs; + if (admission.mode === "refuse" || timedOut) await this.store.release(id); return { ok: false, refusal: { @@ -171,8 +176,8 @@ export abstract class SubstrateFacadeBase extends WorkerEntrypoint implemen poolBusy: admitted.poolBusy, cap: this.caps[pool], position: admitted.position, - queuedForMs: Math.max(0, Date.now() - admitted.enqueuedAt), - retryAfterMs: ADMISSION_RETRY_AFTER_MS, + queuedForMs, + ...(timedOut ? { timedOut: true } : { retryAfterMs: ADMISSION_RETRY_AFTER_MS }), }, }; } diff --git a/apps/substrate/src/facade.workers.test.ts b/apps/substrate/src/facade.workers.test.ts index dd4b232..9dcc7c4 100644 --- a/apps/substrate/src/facade.workers.test.ts +++ b/apps/substrate/src/facade.workers.test.ts @@ -113,6 +113,43 @@ describe("admission (ADR-0004) - the ceiling is the substrate's to own", () => { expect(outcome.refusal).toHaveProperty("retryAfterMs"); }); + it("keeps a queued execution's row while it is under its age ceiling", async () => { + await dispatcher().admissionAttempt(freshKey(), RECIPE); + await dispatcher().admissionAttempt(freshKey(), RECIPE); + + const waiting = freshKey(); + const outcome = await dispatcher().ensureSandbox(waiting, RECIPE, { + mode: "queue", + maxQueueAgeMs: 60 * 60_000, + }); + expect(outcome).toMatchObject({ ok: false, refusal: { kind: "admission-refused" } }); + if (outcome.ok) return; + expect(outcome.refusal).toHaveProperty("retryAfterMs"); + expect(outcome.refusal).not.toHaveProperty("timedOut"); + const behind = await dispatcher().admissionEnqueue(freshKey(), RECIPE); + expect(behind.position).toBeGreaterThanOrEqual(1); + }); + + it("times a queued execution out past maxQueueAgeMs and releases its row", async () => { + await dispatcher().admissionAttempt(freshKey(), RECIPE); + await dispatcher().admissionAttempt(freshKey(), RECIPE); + + const before = await dispatcher().poolStatus(); + const queuedBefore = before.pools.find((p) => p.pool === "lean")?.queued ?? 0; + const outcome = await dispatcher().ensureSandbox(freshKey(), RECIPE, { + mode: "queue", + maxQueueAgeMs: 0, + }); + expect(outcome).toMatchObject({ + ok: false, + refusal: { kind: "admission-refused", pool: "lean", timedOut: true }, + }); + if (outcome.ok) return; + expect(outcome.refusal).not.toHaveProperty("retryAfterMs"); + const after = await dispatcher().poolStatus(); + expect(after.pools.find((p) => p.pool === "lean")?.queued).toBe(queuedBefore); + }); + it("partitions the pools by consumer policy, never by anything a caller says", async () => { // fractalbot lands on `task` (cap 1) and the dispatcher on `lean` (cap 2) // from the SAME recipe: the image class is policy-selected from the diff --git a/packages/runtime-cf/src/sandbox-facade.ts b/packages/runtime-cf/src/sandbox-facade.ts index e4064bf..dc8d927 100644 --- a/packages/runtime-cf/src/sandbox-facade.ts +++ b/packages/runtime-cf/src/sandbox-facade.ts @@ -114,7 +114,9 @@ export const describeRefusal: (refusal: SubstrateRefusal) => string = Match.type().pipe( Match.discriminatorsExhaustive("kind")({ "admission-refused": (r) => - `substrate admission refused: pool ${r.pool} is ${r.poolBusy}/${r.cap} busy`, + r.timedOut + ? `substrate admission timed out: queued ${r.queuedForMs ?? 0}ms behind pool ${r.pool} at ${r.poolBusy}/${r.cap} busy` + : `substrate admission refused: pool ${r.pool} is ${r.poolBusy}/${r.cap} busy`, "approval-required": (r) => `substrate refused an irreversible command: the run definition does not pre-assert "${r.rule}"`, "attestation-rejected": (r) => `substrate rejected the approval attestation: ${r.reason}`, diff --git a/packages/substrate-contract/src/index.ts b/packages/substrate-contract/src/index.ts index b513cbb..dc5522e 100644 --- a/packages/substrate-contract/src/index.ts +++ b/packages/substrate-contract/src/index.ts @@ -261,6 +261,12 @@ export type AdmissionRefused = { position?: number; queuedForMs?: number; retryAfterMs?: number; + /** + * `true` when a `{mode:'queue'}` execution has waited `maxQueueAgeMs` or longer + * without admission. The substrate has released its queue row, so polling + * again re-enqueues at the back; stop and report instead. + */ + timedOut?: boolean; }; /** The command matches the irreversible floor and no attestation was carried. */ @@ -373,7 +379,7 @@ export type DetachedProcess = { }; /** - * What a detached process is doing. `unknown` is its own state rather than an + * What a detached process is doing. `gone` is its own state rather than an * error: a container that slept, restarted or was checkpointed no longer has * the process, and a consumer polling from a durable step needs to tell that * apart from "still running" without catching a throw. @@ -491,7 +497,8 @@ export interface SubstrateFacade { * `{mode:'refuse'}` fails fast with `admission-refused` when the pool is * full; `{mode:'queue'}` expects the consumer to have driven * admissionEnqueue/Attempt to admission first, and refuses (never blocks) - * when it has not. + * when it has not. A queued execution that has waited `maxQueueAgeMs` or + * longer is refused with `timedOut: true` and loses its queue row. */ ensureSandbox( key: SandboxKey,