Enforce execution limits with a balance gate and per-org rate-limit backstop#1274
Enforce execution limits with a balance gate and per-org rate-limit backstop#1274RhysSullivan wants to merge 6 commits into
Conversation
Check the org's Autumn execution balance before execute/executeWithPause (never resume, so paused executions can always complete). Blocked orgs get a descriptive ExecuteResult.error instead of running. Fails open on any billing error or a 2s timeout, and caches per-org outcomes for 60s.
1000 execute calls per org per hour, fixed window, counted in a minimal
per-org Durable Object (EXECUTION_RATE_LIMITER, one instance per org via
idFromName). Independent of billing so runaway automation is caught even
when Autumn is down; fails open if the counter DO errors or is slow. The
DO stores a single {windowId, count} record and purges itself by alarm
after two idle windows.
Order: rate-limit backstop first (cheap counter), then the balance gate, then usage tracking. Guards wrap outside the tracker so a blocked execution is neither run nor tracked. Covers both planes (HTTP executor plane and MCP session DO) since both build engines through this layer.
Covers allow/block, the typed errors, fail-open on billing errors and timeouts (asserting the check was attempted AND the execution ran), the 60s per-org outcome cache (allowed and blocked cached, errors never), window rollover, per-org isolation, and that resume is never gated.
Cloudflare preview
Sign-in is Cloudflare Access (one-time PIN to an allowed email). The preview has its own database and encryption key; it is destroyed when this PR closes. |
@executor-js/cli
@executor-js/config
@executor-js/execution
@executor-js/sdk
@executor-js/codemode-core
@executor-js/runtime-quickjs
@executor-js/plugin-file-secrets
@executor-js/plugin-graphql
@executor-js/plugin-keychain
@executor-js/plugin-mcp
@executor-js/plugin-onepassword
@executor-js/plugin-openapi
executor
commit: |
Greptile SummaryThis PR adds two pre-execution guards to
Confidence Score: 3/5Safe to merge after verifying Autumn's check() response for unlimited/non-metered plan customers; all other changes are well-tested and fail open. The core concern is checkExecutionBalance returning check.allowed verbatim from Autumn's check() call. The fail-open path only fires when Autumn throws an error. If Autumn returns {allowed: false} for customers on plans where executions are not metered, those orgs would be silently blocked on deploy. The rate limiter and gate logic themselves are clean, tests are solid, and the Durable Object wiring is correct. apps/cloud/src/extensions/billing/service.ts — the checkExecutionBalance implementation needs verification that Autumn's check() throws (rather than returning {allowed: false}) for customers on unlimited or non-execution-metered plans. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Client as MCP/HTTP Client
participant RL as Rate Limiter (DO)
participant BG as Balance Gate (Autumn, cached 60s)
participant UT as Usage Tracker (fire-and-forget)
participant E as Engine
Client->>RL: execute(code)
alt "count > 1000/hr"
RL-->>Client: ExecuteResult.error (rate limit)
else within limit
RL->>BG: decide(orgId)
alt Autumn error / timeout
BG-->>BG: fail open (warn + Sentry)
BG->>UT: allowed
else "allowed = false"
BG-->>Client: ExecuteResult.error (quota exceeded)
else "allowed = true"
BG->>UT: allowed
end
UT->>E: execute(code)
E-->>UT: result
UT->>UT: Effect.runFork(trackExecution) [async]
UT-->>Client: result
end
Note over Client,E: resume() bypasses both guards entirely
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Client as MCP/HTTP Client
participant RL as Rate Limiter (DO)
participant BG as Balance Gate (Autumn, cached 60s)
participant UT as Usage Tracker (fire-and-forget)
participant E as Engine
Client->>RL: execute(code)
alt "count > 1000/hr"
RL-->>Client: ExecuteResult.error (rate limit)
else within limit
RL->>BG: decide(orgId)
alt Autumn error / timeout
BG-->>BG: fail open (warn + Sentry)
BG->>UT: allowed
else "allowed = false"
BG-->>Client: ExecuteResult.error (quota exceeded)
else "allowed = true"
BG->>UT: allowed
end
UT->>E: execute(code)
E-->>UT: result
UT->>UT: Effect.runFork(trackExecution) [async]
UT-->>Client: result
end
Note over Client,E: resume() bypasses both guards entirely
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const checkExecutionBalance = (organizationId: string) => | ||
| Effect.gen(function* () { | ||
| yield* Effect.annotateCurrentSpan({ "autumn.customer.id": organizationId }); | ||
| const check = yield* use((c) => | ||
| c.check({ customerId: organizationId, featureId: "executions" }), | ||
| ); | ||
| return { allowed: check.allowed }; | ||
| }).pipe(Effect.withSpan("autumn.checkExecutionBalance")); |
There was a problem hiding this comment.
check() for unlimited-plan orgs may silently block executions
checkExecutionBalance returns { allowed: check.allowed } verbatim. The balance gate's fail-open path only triggers on an AutumnError (i.e., when use(...) throws). If Autumn's check() returns { allowed: false } for customers on plans where the executions feature is unlimited or not feature-gated (rather than throwing), those orgs will be blocked without any fail-open. The PR description says it "fails open on missing customer", which implies Autumn throws for unknown customers — but orgs on enterprise or legacy plans that have executions either unconfigured or returned as false would be silently blocked with no warning. Worth verifying the exact Autumn response shape for unlimited/non-metered plan customers before this ships.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
executor-cloud | 3b4b919 | Jul 02 2026, 09:25 PM |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | 3b4b919 | Commit Preview URL Branch Preview URL |
Jul 02 2026, 09:25 PM |
1 participant
Problem
Execution usage is tracked to the billing provider after every execution, but nothing ever checks the balance before running. An org on the free plan (10k executions/month, no overage) can keep executing indefinitely past its quota: the ledger clamps at the cap while executions continue at full speed. Observed in production as one tenant sustaining automated polling around the clock, far past its included usage, for over a week.
What this adds
Two pre-execution guards in
CloudMeteringEngineDecorator, the one decorator both cloud planes (MCP session DO and HTTP executor plane) build engines through. They wrap outside usage tracking, so a blocked execution is neither run nor billed.resumeis never gated: a paused execution already consumed its slot, and blocking resume would strand approved work.1. Balance gate (
execution-gate.ts)executionsfeature via the billing service beforeexecute/executeWithPause.trackExecution.ExecuteResult.error(the same channel compilation errors use, since engine error-channel failures are deliberately rendered opaque by the host): a cleanisErrortool result telling the user their plan's included executions are used up.2. Rate-limit backstop (
execution-rate-limit.ts)idFromName(orgId), single{windowId, count}record, alarm-purged after two idle windows). NewEXECUTION_RATE_LIMITERbinding + migrationv3.Tests
19 unit tests across both guards: allow/block paths, fail-open on error and on timeout, cache TTL and per-org isolation, error outcomes never cached, window reset, and resume never gated. Fail-open tests assert the check was attempted AND the execution ran.
Verification
bun run typecheck— 42/42 greenvitest run src/engine/execution-gate.test.ts src/engine/execution-rate-limit.test.ts— 19/19bun run lint,bun run format:check— cleanDeploy notes
v3creates the counter DO class; standard single deploy. In workers without the binding (tests, older local setups) the limiter logs a warning and disables itself.