Skip to content

test(host-selfhost): raise scope-isolation timeout to 120s#1252

Open
RhysSullivan wants to merge 1 commit into
mainfrom
deflake-scope-isolation
Open

test(host-selfhost): raise scope-isolation timeout to 120s#1252
RhysSullivan wants to merge 1 commit into
mainfrom
deflake-scope-isolation

Conversation

@RhysSullivan

Copy link
Copy Markdown
Owner

Summary

scope-isolation.test.ts > concurrent requests with distinct identities get disjoint, correct executor bindings has been timing out at the default 30s on recent CI runs — it failed the Test job on #1247's and #1248's runs (both unrelated changes) and reproduces on pristine main locally about 1 in 3 runs under CPU load.

Where the time goes

Instrumented locally: the timeout is hit during seeding, before the concurrency under test even starts. Seeding is 6 sequential addSpec calls, and a single addSpec ranges from ~0.5s to ~15s when the vitest fork pool oversubscribes the CPU — every host-selfhost test file boots a full app at import and all queries serialize through one libSQL connection. This is the same mechanism #1208 identified; that PR bounded the test's own concurrency (48-wide burst → 6) but left the default 30s ceiling, which the seeding phase alone can blow through. A passing run under load was observed at 58s wall clock.

Fix

An explicit 120s timeout with a comment explaining why. The test asserts isolation, not latency — a real request-scoped-binding leak fails on correctness assertions immediately regardless of the clock, so a generous bound loses nothing.

Verified: 5 consecutive local runs pass (22–60s range under load), full host-selfhost suite green (72/72).

The concurrent-identities test has been timing out at the default 30s on
recent CI runs (and locally under load, ~1 in 3). The time goes into
seeding: 6 sequential addSpec calls, each 0.5-15s when the vitest fork
pool oversubscribes the CPU (every host-selfhost test file boots a full
app and serializes queries through one libSQL connection - the same
mechanism #1208 addressed). Locally the test body was observed at 58s
wall clock while still passing every assertion.

The test asserts isolation, not latency, so a generous explicit timeout
is the honest bound: a real binding leak still fails on correctness
immediately.
@greptile-apps

greptile-apps Bot commented Jul 2, 2026

Copy link
Copy Markdown

Greptile Summary

This PR raises the Vitest timeout for the concurrent requests with distinct identities get disjoint, correct executor bindings test from 30s to 120s, and adds an inline comment documenting why. The root cause is that the seeding phase (6 sequential addSpec calls) can each take up to ~15s under fork-pool CPU oversubscription, blowing through the 30s ceiling before the actual concurrency assertions even begin.

  • The correctness assertion (isolation between request-scoped bindings) is unchanged; only the time budget is widened.
  • A clear comment at the timeout site explains the CPU-oversubscription mechanism so future maintainers know why 120s is intentional.

Confidence Score: 5/5

Safe to merge — only the time budget for one flaky test changes; all correctness assertions are untouched.

The change is a single-line timeout bump with an explanatory comment. The isolation assertions themselves are unchanged, and the 120s ceiling is well-justified by the measured 22–60s wall-clock range under load. No logic, API surface, or data path is affected.

No files require special attention.

Important Files Changed

Filename Overview
apps/host-selfhost/src/scope-isolation.test.ts Timeout raised from 30_000 to 120_000ms with an explanatory comment; no logic or assertion changes.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Vitest as Vitest (fork pool)
    participant Test as scope-isolation.test.ts
    participant Handler as App Handler
    participant DB as libSQL (single connection)

    Note over Vitest,DB: Seeding phase (sequential, up to 6 × 15s under CPU load)
    loop 6 identities
        Test->>Handler: POST /api/openapi/specs (addSpec)
        Handler->>DB: serialize write
        DB-->>Handler: ok
        Handler-->>Test: 200
        Test->>Handler: POST /api/connections
        Handler->>DB: serialize write
        DB-->>Handler: ok
        Handler-->>Test: 200
    end

    Note over Vitest,DB: Concurrency phase (8 rounds × 6 concurrent reads)
    loop 8 rounds
        par 6 concurrent identities
            Test->>Handler: GET /api/connections (identity N)
            Handler->>DB: read
            DB-->>Handler: rows
            Handler-->>Test: 200 + addresses
        end
    end

    Note over Test: Assert each identity sees only its own connection (isolation check)
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Vitest as Vitest (fork pool)
    participant Test as scope-isolation.test.ts
    participant Handler as App Handler
    participant DB as libSQL (single connection)

    Note over Vitest,DB: Seeding phase (sequential, up to 6 × 15s under CPU load)
    loop 6 identities
        Test->>Handler: POST /api/openapi/specs (addSpec)
        Handler->>DB: serialize write
        DB-->>Handler: ok
        Handler-->>Test: 200
        Test->>Handler: POST /api/connections
        Handler->>DB: serialize write
        DB-->>Handler: ok
        Handler-->>Test: 200
    end

    Note over Vitest,DB: Concurrency phase (8 rounds × 6 concurrent reads)
    loop 8 rounds
        par 6 concurrent identities
            Test->>Handler: GET /api/connections (identity N)
            Handler->>DB: read
            DB-->>Handler: rows
            Handler-->>Test: 200 + addresses
        end
    end

    Note over Test: Assert each identity sees only its own connection (isolation check)
Loading

Reviews (1): Last reviewed commit: "test(host-selfhost): raise scope-isolati..." | Re-trigger Greptile

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
executor-marketing 78f60c7 Commit Preview URL

Branch Preview URL
Jul 02 2026, 02:14 AM

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
executor-cloud 78f60c7 Jul 02 2026, 02:14 AM

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Cloudflare preview

Console https://executor-preview-pr-1252.executor-e2e.workers.dev
MCP https://executor-preview-pr-1252.executor-e2e.workers.dev/mcp
Deployed commit 78f60c7

Sign-in is Cloudflare Access (one-time PIN to an allowed email). The preview has its own database and encryption key; it is destroyed when this PR closes.

@pkg-pr-new

pkg-pr-new Bot commented Jul 2, 2026

Copy link
Copy Markdown

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@1252

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@1252

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@1252

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@1252

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@1252

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@1252

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@1252

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@1252

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@1252

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@1252

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@1252

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@1252

executor

npm i https://pkg.pr.new/executor@1252

commit: 78f60c7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant