Vitest integration: use --vitest-config instead of --command to prevent wrong files selection

[anmarchenko]

What

Vitest workers now execute only their assigned files through a directly launched Node API adapter. Discovery and execution load the same Vitest config. Vitest 3–5 use the public specification API; a separate module handles Vitest 1.6–2.

The adapter creates a non-watch context, discovers specifications, keeps the exact assigned paths, runs those specifications, and shuts down through Vitest's exit() API, honoring teardownTimeout when leaked handles remain. It preserves multiple project/pool specifications for an assigned file. The previous CLI interception, adapter preload, argument rewriting, and CLI discovery fallback are removed.

Vitest integration changes: install Vitest locally and use dd-trace 5.125.0 or higher. --command is rejected for Vitest with migration instructions. Put Vitest options in its config and use the new --vitest-config / DD_TEST_OPTIMIZATION_RUNNER_VITEST_CONFIG setting for a non-default file. Run shell/npm preparation steps before ddtest. The migration guide covers project filters, reporters, coverage, snapshots, Node options, and unsupported CLI workflows.

Why

With test.dir: 'src', the filename filter for src/endOfYear/test.ts also matches src/eachWeekendOfYear/test.ts. Disjoint worker assignments can therefore execute a file twice or run a file excluded from the plan.

Vitest documents substring filename filtering. Its programmatic lifecycle and specification API let ddtest run exact assignments while Vitest handles projects, setup/teardown, reporters, and coverage. Direct invocation gives this integration an explicit configuration contract. dd-trace-js #10161, released in 5.125.0, instruments createVitest directly.

Closes #168.

E2E testing

Prerequisites: check out this branch; use the Go version in go.mod, Node 22.14.0 or later on PATH, npm, Git, and npm registry access. These scenarios disable tracing and need no Datadog credentials. To verify reporting in Datadog, repeat the passing/failing scenarios in an environment with Test Optimization configured and confirm the assigned tests and completed session appear.

1. Build the binary and prepare an isolated fixture:

   go build -o /tmp/ddtest-issue-168 main.go
   fixture_dir="$(mktemp -d)"
   cd "$fixture_dir"
   git init -q
   git -c user.name=Reproducer -c user.email=reproducer@example.invalid \
     -c commit.gpgsign=false commit --allow-empty -qm 'Initialize fixture'
   cat > package.json <<'JSON'
   {"private":true,"type":"module","devDependencies":{"vitest":"5.0.0","dd-trace":"5.125.0"}}
   JSON
   npx --yes npm@11.11.1 install
   cat > vitest.ci.config.mjs <<'JS'
   export default { test: { dir: 'src', include: ['**/test.ts'] } }
   JS
   mkdir -p src/endOfYear src/eachWeekendOfYear
   cat > src/record.mjs <<'JS'
   import { appendFileSync, writeFileSync } from 'node:fs'
   export function record(name) {
     appendFileSync(process.env.REPRO_EVENTS, JSON.stringify({
       file: name, session: process.env.DD_TEST_SESSION_NAME || 'direct',
     }) + '\n')
     if (process.env.REPRO_ONCE === '1')
       writeFileSync(`.created-${name}`, 'created', { flag: 'wx' })
   }
   JS
   cat > src/endOfYear/test.ts <<'JS'
   import { test, expect } from 'vitest'
   import { record } from '../record.mjs'
   test('endOfYear', () => { record('endOfYear'); expect(1).toBe(1) })
   JS
   cat > src/eachWeekendOfYear/test.ts <<'JS'
   import { test, expect } from 'vitest'
   import { record } from '../record.mjs'
   test('eachWeekendOfYear', () => { record('eachWeekendOfYear'); expect(1).toBe(1) })
   JS
   export DD_API_KEY='' DD_APP_KEY='' NODE_OPTIONS=''
   export DD_CIVISIBILITY_ENABLED=false DD_TRACE_ENABLED=false
   export DD_TRACE_PACKAGE="$PWD/node_modules/dd-trace/ci/init"
   export DD_TRACE_ESM_IMPORT="$PWD/node_modules/dd-trace/register.js"
   export REPRO_EVENTS="$PWD/manual-events.jsonl" REPRO_ONCE=0

   Expected: two passing test files under the configured src directory. Perform the remaining actions inside this fixture.

2. Compare native CLI filtering with exact ddtest execution:

   node node_modules/vitest/vitest.mjs run --config vitest.ci.config.mjs src/endOfYear/test.ts
   : > "$REPRO_EVENTS"
   export REPRO_ONCE=1
   /tmp/ddtest-issue-168 run --platform javascript --framework vitest \
     --vitest-config vitest.ci.config.mjs --min-parallelism 2 --max-parallelism 2
   cat "$REPRO_EVENTS"
   cat .testoptimization/runner/tests-split/*

   Expected: native Vitest runs both files. DDTest exits zero, records each file exactly once under different worker sessions, and creates one resource file per test without an EEXIST error. Saved split files show disjoint assignments. All processes finish after the run.

3. Verify wrapper migration fails clearly, then use the environment config setting:

   export REPRO_ONCE=0
   rm -rf .testoptimization
   : > "$REPRO_EVENTS"
   /tmp/ddtest-issue-168 run --platform javascript --framework vitest \
     --command 'npx vitest run --config vitest.ci.config.mjs'

   Expected: nonzero exit with instructions to remove --command and use --vitest-config; no test events. Then run:

   export DD_TEST_OPTIMIZATION_RUNNER_VITEST_CONFIG=vitest.ci.config.mjs
   /tmp/ddtest-issue-168 plan --platform javascript --framework vitest \
     --min-parallelism 1 --max-parallelism 1 -- src/endOfYear/test.ts
   /tmp/ddtest-issue-168 run --platform javascript --framework vitest
   cat "$REPRO_EVENTS"

   Expected: planning loads the selected config without executing tests; execution records only endOfYear. Keep this config environment variable set for the following steps.

4. Verify exclusions and assigned failures:

   rm -rf .testoptimization
   : > "$REPRO_EVENTS"
   /tmp/ddtest-issue-168 run --platform javascript --framework vitest \
     --min-parallelism 1 --max-parallelism 1 \
     --tests-exclude-pattern src/eachWeekendOfYear/test.ts
   cat "$REPRO_EVENTS"

   Expected: exit zero and one endOfYear event. Change its assertion to expect(1).toBe(2), clear the saved plan/events, and repeat: expect nonzero exit and still no excluded file execution. Restore the passing assertion.

5. Verify configured project selection and JSON reporting on Vitest 5.0.0:

   cat > projects.config.mjs <<'JS'
   export default { test: {
     dir: 'src', include: ['**/test.ts'], project: ['unit'],
     reporters: ['json'], outputFile: 'qa-report.json',
     projects: [
       { extends: true, test: { name: 'unit' } },
       { extends: true, test: { name: 'integration' } },
     ],
   } }
   JS
   rm -rf .testoptimization
   : > "$REPRO_EVENTS"
   /tmp/ddtest-issue-168 run --platform javascript --framework vitest \
     --vitest-config projects.config.mjs --min-parallelism 1 --max-parallelism 1 \
     -- src/endOfYear/test.ts
   cat "$REPRO_EVENTS"
   cat qa-report.json

   Expected: one endOfYear event and one passing test in the JSON report. Remove project: ['unit'], clear the plan/events and repeat: expect two endOfYear events and two passing tests, one per project, with no eachWeekendOfYear execution.

6. Verify CI snapshots and configured sharding:

   mkdir -p src/snapshot
   cat > src/snapshot/test.ts <<'JS'
   import { test, expect } from 'vitest'
   test('snapshot policy', () => { expect({ assigned: true }).toMatchSnapshot() })
   JS
   cat > snapshots.config.mjs <<'JS'
   export default { test: {
     dir: 'src', include: ['**/test.ts'], shard: '1/1',
     update: process.env.REPRO_UPDATE === '1',
   } }
   JS
   rm -rf .testoptimization
   CI=true /tmp/ddtest-issue-168 run --platform javascript --framework vitest \
     --vitest-config snapshots.config.mjs --min-parallelism 1 --max-parallelism 1 \
     -- src/snapshot/test.ts

   Expected: missing-snapshot failure and no snapshot file, with no watch-mode/sharding error. Clear the saved plan and repeat with CI=true REPRO_UPDATE=1: expect exit zero and src/snapshot/__snapshots__/test.ts.snap containing the assigned value.

7. Verify shutdown after completed teardown with a leaked timer:

   cat > shutdown.config.mjs <<'JS'
   export default { test: {
     dir: 'src', include: ['endOfYear/test.ts'],
     globalSetup: ['./shutdown-setup.mjs'], teardownTimeout: 1000,
   } }
   JS
   cat > shutdown-setup.mjs <<'JS'
   export default function () {
     setInterval(() => {}, 1000)
     return async () => {
       await new Promise(resolve => setTimeout(resolve, 50))
       console.log('QA_TEARDOWN_COMPLETED')
     }
   }
   JS
   rm -rf .testoptimization
   time /tmp/ddtest-issue-168 run --platform javascript --framework vitest \
     --vitest-config shutdown.config.mjs --min-parallelism 1 --max-parallelism 1

   Expected: the assigned test passes, QA_TEARDOWN_COMPLETED appears, and the process exits zero after Vitest's one-second shutdown timeout instead of hanging. Vitest prints its timeout diagnostic because the interval is still active. Change the assertion to expect(1).toBe(2), clear the plan, and repeat: teardown still completes and shutdown remains bounded, with a nonzero exit. Restore the passing assertion. Repeat both runs with tracing enabled in a configured Test Optimization environment: confirm the assigned test, completed suite, and completed session arrive with the correct pass/fail status.

8. Repeat steps 2–4 and 7 on Vitest 1.6.1, 2.1.9, 3.2.7, and 4.1.11, installing each with npx --yes npm@11.11.1 install --save-dev vitest@VERSION. Before each version, restore passing assertions, remove src/snapshot, clear .testoptimization and event records, and remove .created-endOfYear / .created-eachWeekendOfYear. Expected: exact assignments, config selection, exclusions, failure status, bounded shutdown, and wrapper migration behave the same on every version.

9. Leave the fixture, then remove it and the temporary binary:

   cd /tmp
   rm -rf "$fixture_dir"
   rm -f /tmp/ddtest-issue-168

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T15:04:37.504418Z	faf2310	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[datadog-datadog-prod-us1-2]

 

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 89.29%
• Overall Coverage: 92.28% (+0.03%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4b69fe7 | Docs | View more details | Give us feedback!}

[datadog-datadog-prod-us1-2]

More details

The static review raised no reportable concern with the exact-file filtering hook. Supported-version Vitest integration remains unverified locally because dependencies were unavailable.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit faf2310 · @DataDog review to ask questions}

[anmarchenko]

@autotest review

[datadog-datadog-prod-us1-2]

Forced watch mode can make missing CI snapshots pass and prevents sharded configurations from initializing. The version gate also selects an unavailable execution API on Vitest 3.0–3.1.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 2db1a00 · @DataDog review to ask questions}

[anmarchenko]

E2E verification: passed

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)
Environment: Local macOS arm64; Node 24.14.0, Go 1.27.1, dd-trace 5.125.0
Commit tested: 29fc98776441eb83826f5dc9cbc0d1ddbb60ac08

The original duplicate-file bug is fixed, and the leaked-timer shutdown regression found during the initial review is now fixed. Shepherd playgrounds require migration to the new config-based interface; the migration is prepared in https://github.com/ddoghq/shepherd/pull/154.

Check	Result
Adapter and tracing compatibility	Passed on Vitest 1.6.1, 2.1.9, 3.2.7, 4.1.11, and 5.0.0, with test caching disabled
Leaked timer after completed async teardown	Passing and failing CLI runs exit with the correct status on all five versions; no watchdog termination
Traced shutdown on Vitest 4 and 5	Exactly one test, suite, module, and completed session per run, with correct pass/fail statuses
Duplicate-file reproducer	Base executes three tests for two assignments; fixed PR executes exactly two
CLI selection and failure behavior	Saved-plan selection, config environment selection, exclusions, wrapper rejection, and assigned-test failure checks passed
Migrated Vue parallel playground	All 33 assertions passed: 3,558 tests, 182 suites, two modules, and two completed sessions
Vue assignment completeness	182 unique emitted filenames exactly match the saved plan; zero duplicate files
Repository checks	make test and make lint passed; zero lint issues

Issues and resolution

The initial revision 35ad6375a4aec91044ba098c0eb91b240ac8260d hung on Vitest 4.1.11 and 5.0.0 when global setup leaked an interval, even after async teardown completed. The same reproducer still exceeds an eight-second watchdog on that revision. At the tested head, bounded shutdown completes in 1.78–3.04 seconds without tracing, preserving exit 0 for passing tests and exit 1 for failing tests. Additional traced runs finish in 3.67–4.06 seconds, including flushing. The new regression tests also pass.

The new interface rejects the old --command wrapper and environment override, requires locally installed Vitest, and requires dd-trace 5.125.0 or later for the API integration. Shepherd's migration moves Vitest options into a shared config selected through --vitest-config, preserving the Vue unit projects. The migrated full Vue workload passed against this head.

Telemetry verification and methodology

Datadog UI verification: Production Datadog UI ingestion was not tested. Actual tracer events received by Mockdog were parsed and checked for counts, completion, filenames, and statuses.

Validation combined fresh CLI fixtures with exact saved assignments, an uncleared interval and completed async teardown, a bounded external watchdog, separate Mockdog instances for traced pass/fail cases, the adapter/tracing compatibility matrix, and Shepherd's full migrated Vue parallel workload. The PR head was rechecked after testing and still points to the commit above.

[anmarchenko]

@autotest review

[datadog-datadog-prod-us1-2]

More details

Exact-path filtering preserves each assigned file’s project/pool specifications while avoiding Vitest’s substring filename matching. Runtime compatibility across supported Vitest versions remains unverified locally because dependencies were unavailable.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 29fc987 · @DataDog review to ask questions}