Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -270,9 +270,10 @@ jobs:
vitest_dir="${RUNNER_TEMP}/vitest-${{ matrix.vitest }}"
# npm 10.9.8 crashes while resolving Vitest's peer dependencies.
# Keep peer resolution enabled: Vitest 5 requires Vite as a peer.
npx --yes npm@11.11.1 install --prefix "${vitest_dir}" "vitest@${{ matrix.vitest }}"
npx --yes npm@11.11.1 install --prefix "${vitest_dir}" "vitest@${{ matrix.vitest }}" "@vitest/coverage-v8@${{ matrix.vitest }}" "dd-trace@5.125.0"
DDTEST_VITEST_NODE_MODULES="${vitest_dir}/node_modules" \
go test -v ./internal/compatibility -run '^TestVitestAdapterIntegration$'
DDTEST_DD_TRACE_NODE_MODULES="${vitest_dir}/node_modules" \
go test -v ./internal/compatibility -run '^TestVitest(Adapter|Tracing)Integration$'

mocha-compatibility:
runs-on: ubuntu-latest
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ Minimum supported library and runtime requirements:
Cucumber support is tested with `@cucumber/cucumber` 7 through 13; Cypress
support requires Cypress 12 or higher; Mocha support requires Mocha 8 or higher;
Playwright support requires Playwright 1.18 or higher; Vitest support requires
Vitest 1.6 or higher.
Vitest 1.6 or higher and `dd-trace` **5.125.0** or higher. Vitest uses a direct
Node API integration; see [configuration and migration instructions](docs/running.md#vitest-integration).

For instructions on setting up Test Optimization, see the [Datadog Test Optimization documentation](https://docs.datadoghq.com/tests/setup/).

Expand Down Expand Up @@ -216,7 +217,8 @@ parallelism details, see [Running DDTest](docs/running.md).
| --- | --- |
| `--platform` | Language/platform. Currently supported: `ruby`, `python`, `javascript`. |
| `--framework` | Test framework. Currently supported: `rspec`, `minitest`, `pytest`, `cucumber`, `cypress`, `jest`, `mocha`, `playwright`, `vitest`. |
| `--command` | Override the default base command for supported framework modes. Used by RSpec and Minitest run/discovery, Cucumber, Cypress, Jest, Mocha, Playwright, and Vitest run/discovery, and pytest run/discovery (since 1.7.0). For ddtest versions prior to 1.7.0 with pytest, the command cannot be changed. Pass extra flags with `PYTEST_ADDOPTS`. |
| `--command` | Override the default base command for supported framework modes. Used by RSpec and Minitest run/discovery, Cucumber, Cypress, Jest, Mocha, and Playwright run/discovery. Vitest rejects this option; use `--vitest-config` and configure options in Vitest. |
| `--vitest-config` | Vitest config file used by both planning and execution; defaults to Vitest config discovery. See [migration instructions](docs/running.md#vitest-integration). |
| `--min-parallelism` | Minimum CI node or worker count DDTest considers when planning. |
| `--max-parallelism` | Maximum CI node or worker count DDTest considers when planning. |
| `--target-time` | Target wall time DDTest tries to satisfy when selecting parallelism. |
Expand Down
15 changes: 8 additions & 7 deletions docs/best_practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,17 +170,18 @@ file list and Jest flags itself.

## Vitest Support

Use `--command` when your project runs Vitest through a package manager or uses
Vitest projects:
Install Vitest and `dd-trace` 5.125.0 or higher in the project. Put Vitest options
in its configuration; DDTest uses the Node API directly and rejects `--command`.
Select a non-default configuration consistently during planning and execution:

```bash
ddtest run --platform javascript --framework vitest --command "pnpm exec vitest run --project unit*"
ddtest plan --platform javascript --framework vitest --vitest-config vitest.ci.config.ts
ddtest run --platform javascript --framework vitest --vitest-config vitest.ci.config.ts
```

The command must invoke Vitest directly. During planning, DDTest changes the
`run` subcommand to `list --filesOnly --json` on Vitest 2.0 and newer. On Vitest 1.6,
DDTest passes the Vitest arguments to its config-aware discovery API. DDTest
supplies the selected test files during execution.
Run package-script setup steps before DDTest and export any required environment
variables. See [Vitest integration](running.md#vitest-integration) for migrating
command flags, selecting projects, and configuring reporters and coverage.

## Mocha Support

Expand Down
101 changes: 70 additions & 31 deletions docs/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ starting each worker.
Use `--command` to override the framework's default base test command where
supported. DDTest applies this override to RSpec run and full
discovery, Minitest run and full discovery, Cucumber, Cypress, Jest, Mocha,
Playwright, and Vitest run and file discovery, and pytest run and discovery
and Playwright run and file discovery, and pytest run and discovery
(since 1.7.0):

```bash
Expand All @@ -201,13 +201,9 @@ replaces configured `spec` entries with each worker's assigned files:
ddtest run --platform javascript --framework mocha --command "pnpm exec mocha --parallel"
```

For JavaScript/Vitest, the command must invoke Vitest directly. During planning,
DDTest uses `list --filesOnly --json` on Vitest 2.0 and newer and the config-aware
discovery API on Vitest 1.6. It appends selected files during execution:

```bash
ddtest run --platform javascript --framework vitest --command "pnpm exec vitest run --project unit*"
```
Vitest uses a direct Node API integration and does not accept `--command`.
Use `--vitest-config` to select a configuration file; see
[Vitest integration](#vitest-integration) for migration instructions.

For JavaScript/Cypress, the command must invoke Cypress directly. DDTest keeps
configuration options such as `--project`, `--config-file`, `--config`,
Expand Down Expand Up @@ -358,35 +354,78 @@ expects Mocha to be resolvable from the current project. Discovery removes
the Datadog CI preload from `NODE_OPTIONS`; test runs retain it for Test
Optimization instrumentation.

## Vitest Discovery And Instrumentation
## Vitest Integration

For JavaScript/Vitest 2.0 or higher, DDTest discovers test files with Vitest's
native `list --filesOnly --json` command. It uses this priority:
DDTest launches its own Node adapter using the project's installed `vitest/node`
API. Install Vitest **1.6 or higher**, `dd-trace` **5.125.0 or higher**, and any
configured coverage or environment packages before running DDTest. Run from the
project directory where Node can resolve those packages. DDTest does not install
Vitest through `npx`.

1. `--command` when set, replacing its Vitest subcommand with `list` and
appending `--filesOnly --json`.
2. The local executable `node_modules/.bin/vitest` when present.
3. `npx vitest`.
Both planning and execution load the normal Vitest/Vite configuration. To use a
non-default file, pass the same `--vitest-config` value to both commands:

Vitest resolves its own Vite/Vitest configuration, projects, and default test
matching. When `--tests-location` or `--tests-exclude-pattern` is set, DDTest
filters the file list returned by Vitest after discovery.

Vitest 1.6 does not support `list --filesOnly`. When DDTest detects that specific
unsupported-option error, it uses the `vitest/node` discovery API instead. This
loads the project's Vitest configuration and discovers files for its configured
projects, include and exclude patterns, and CLI filters without executing tests.
If that API is unavailable, DDTest falls back to its own filesystem glob using
`--tests-location` or the default Vitest test-file pattern.
```bash
ddtest plan --platform javascript --framework vitest --vitest-config vitest.ci.config.ts
ddtest run --platform javascript --framework vitest --vitest-config vitest.ci.config.ts
```

DDTest adds the CI require described above and a `--import` for Vitest worker
processes. It preserves an existing Datadog register import; otherwise it uses
Alternatively, set
`DD_TEST_OPTIMIZATION_RUNNER_VITEST_CONFIG=vitest.ci.config.ts` for both steps.
Paths are relative to the working directory. The option selects a config file;
it does not change the working directory.

Vitest owns configuration, project discovery, setup/teardown, reporters, and
coverage. DDTest runs once with `run: true` and `watch: false`, and executes only
the discovered project/pool specifications whose canonical file paths belong to
the worker's assignment. The same file can still run in multiple configured
projects. Vitest 3–5 use the public specification API; Vitest 1.6–2 use a separate
legacy adapter. Configuration and discovery errors fail the operation rather
than falling back to a filesystem glob.

### Migrating from a Vitest command wrapper

`--command` and `DD_TEST_OPTIMIZATION_RUNNER_COMMAND` are rejected for Vitest,
including direct `vitest`, `node .../vitest.mjs`, `npx`, `pnpm`, and npm-script
invocations. Remove the command override and move Vitest options to its config:

| Previous command option | Configuration or replacement |
| --- | --- |
| `--config vitest.ci.config.ts` | DDTest's `--vitest-config vitest.ci.config.ts` |
| `--project 'unit*'` | `test.project: ['unit*']` |
| `--testNamePattern smoke` | `test.testNamePattern: 'smoke'` |
| `--reporter json --outputFile results.json` | `test.reporters: ['json']`, `test.outputFile: 'results.json'` |
| `--coverage` | `test.coverage.enabled: true` plus an installed coverage provider |
| `--passWithNoTests` | `test.passWithNoTests: true` |
| Test-file arguments | Positional selections to `ddtest plan`, then `ddtest run` |

For example, replace `--command 'pnpm exec vitest run --config
vitest.ci.config.ts --project unit*'` with `--vitest-config vitest.ci.config.ts`
and add `project: ['unit*']` under that file's `test` configuration. Regenerate
any saved plan after changing the configuration or selection.

Shell setup and npm lifecycle scripts are not executed by the adapter. Run
preparation steps before DDTest, and export required environment variables to
DDTest. Keep Node flags and project loaders in `NODE_OPTIONS`; DDTest preserves
them. For Yarn Plug'n'Play, make its loaders available to Node in that environment.
Choose the Node version on `PATH` before starting DDTest.

Use Vitest directly for watch mode, UI, benchmarks, report merging, and other
CLI-only workflows. For snapshot updates, run Vitest with `--update`, or
explicitly enable `test.update` in a dedicated config. Missing snapshots in CI
still fail by default. Prefer DDTest's worker/CI-node settings for splitting;
a configured Vitest `test.shard` additionally shards each assigned batch.

### Tracing

DDTest adds the Datadog CI require and register import to worker `NODE_OPTIONS`.
It preserves an existing Datadog register import; otherwise it uses
`DD_TRACE_ESM_IMPORT`, the `register.js` next to an external tracer's `ci`
directory, or the project-local `dd-trace/register.js`, in that order.
The GitHub action exports the paths, so manually setting `NODE_OPTIONS` is
unnecessary.
The tracer's `--require` option follows existing project loaders. Discovery
removes both Datadog options to avoid instrumenting the file-listing process.
The GitHub action exports the paths, so manually setting Datadog `NODE_OPTIONS`
is unnecessary. The tracer's require follows existing project loaders.
Discovery removes both Datadog options to avoid instrumenting file listing.
The adapter itself is a directly executed script, not a `NODE_OPTIONS` preload.

## Cypress Discovery And Instrumentation

Expand Down
3 changes: 2 additions & 1 deletion docs/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ platforms or frameworks require explicit selection.
| --- | --- | --- | ---: | --- |
| `--platform` | `DD_TEST_OPTIMIZATION_RUNNER_PLATFORM` | | auto-detected | Language/platform. Currently supported: `ruby`, `python`, `javascript`. |
| `--framework` | `DD_TEST_OPTIMIZATION_RUNNER_FRAMEWORK` | | auto-detected | Test framework. Currently supported: `rspec`, `minitest`, `pytest`, `cucumber`, `cypress`, `jest`, `mocha`, `playwright`, `vitest`. |
| `--command` | `DD_TEST_OPTIMIZATION_RUNNER_COMMAND` | | `""` | Override the default base test command for supported framework modes. Used by RSpec and Minitest run/discovery, Cucumber, Cypress, Jest, Mocha, Playwright, and Vitest run/discovery, and pytest run/discovery (since 1.7.0). DDTest appends selected tests and framework-specific flags. For ddtest versions prior to 1.7.0 with pytest, the command cannot be changed. Pass extra flags with `PYTEST_ADDOPTS`. |
| `--command` | `DD_TEST_OPTIMIZATION_RUNNER_COMMAND` | | `""` | Override the default base test command for supported framework modes. Used by RSpec and Minitest run/discovery, Cucumber, Cypress, Jest, Mocha, and Playwright run/discovery. DDTest appends selected tests and framework-specific flags. Vitest rejects this option; use `--vitest-config` and configure options in Vitest. |
| `--vitest-config` | `DD_TEST_OPTIMIZATION_RUNNER_VITEST_CONFIG` | | `""` | Vitest config file used by both planning and execution; defaults to Vitest config discovery. See [migration instructions](running.md#vitest-integration). |
| `--min-parallelism` | `DD_TEST_OPTIMIZATION_RUNNER_MIN_PARALLELISM` | | physical CPU count | Minimum count DDTest considers when planning. Interpret it as CI nodes in CI-node mode, or workers in a single-node run. |
| `--max-parallelism` | `DD_TEST_OPTIMIZATION_RUNNER_MAX_PARALLELISM` | | physical CPU count | Maximum count DDTest considers when planning. Interpret it as CI nodes in CI-node mode, or workers in a single-node run. |
| `--ci-job-overhead` | `DD_TEST_OPTIMIZATION_RUNNER_CI_JOB_OVERHEAD` | | `25s` | Modeled overhead for adding one more CI node. Accepts durations such as `25s`, `1m`, `1500ms`, or `0s` to disable this bias. Increase it to use fewer CI nodes; decrease it to prefer faster wall time. |
Expand Down
14 changes: 9 additions & 5 deletions docs/third-party-runners.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,18 @@ fi

## Vitest

When another runner consumes DDTest's file list for Vitest, load both dd-trace
initialization entry points:
Vitest's CLI treats file arguments as substring filters. Passing a DDTest file
list to `vitest run` can execute additional files with overlapping names.
Use [DDTest's Vitest integration](running.md#vitest-integration) to execute exact
assignments, or build the external runner around Vitest's
[specification API](https://vitest.dev/api/advanced/vitest#runtestspecifications)
and retain only specifications whose file paths are in the assignment.

An external Node API runner must also load both Datadog initialization entry
points and use `dd-trace` 5.125.0 or higher:

```bash
export NODE_OPTIONS="--import dd-trace/register.js -r dd-trace/ci/init${NODE_OPTIONS:+ $NODE_OPTIONS}"
if [ -s .testoptimization/runner/test-files.txt ]; then
xargs ./node_modules/.bin/vitest run < .testoptimization/runner/test-files.txt
fi
```

## Mocha
Expand Down
2 changes: 2 additions & 0 deletions internal/cmd/cmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ var rootPersistentFlagBindings = []persistentFlagBinding{
{configKey: "ci_node", flagName: "ci-node"},
{configKey: "ci_node_workers", flagName: "ci-node-workers"},
{configKey: "command", flagName: "command"},
{configKey: "vitest_config", flagName: "vitest-config"},
{configKey: "tests_location", flagName: "tests-location"},
{configKey: "tests_exclude_pattern", flagName: "tests-exclude-pattern"},
{configKey: "test_discovery_cache", flagName: "test-discovery-cache"},
Expand All @@ -140,6 +141,7 @@ func init() {
rootCmd.PersistentFlags().Int("ci-node", -1, "CI node index to run (0-indexed; default: -1 disables CI-node mode)")
rootCmd.PersistentFlags().String("ci-node-workers", "1", `Number of parallel workers per CI node (positive integer or "ncpu"; default: 1)`)
rootCmd.PersistentFlags().String("command", "", "Test command that ddtest should wrap")
rootCmd.PersistentFlags().String("vitest-config", "", "Vitest configuration file for discovery and execution (defaults to Vitest config discovery)")
rootCmd.PersistentFlags().String("tests-location", "", "Glob pattern used to discover test files")
rootCmd.PersistentFlags().String("tests-exclude-pattern", "", "Glob pattern used to exclude test files from discovery")
rootCmd.PersistentFlags().String("test-discovery-cache", "", "Path to a restored test discovery cache file to import before planning")
Expand Down
7 changes: 7 additions & 0 deletions internal/cmd/cmd_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -651,6 +651,13 @@ func TestFlagBinding(t *testing.T) {
t.Fatalf("Error setting target-time flag: %v", err)
}

if err := rootCmd.PersistentFlags().Set("vitest-config", "config/vitest.ts"); err != nil {
t.Fatal(err)
}
if viper.GetString("vitest_config") != "config/vitest.ts" {
t.Fatalf("vitest-config was not bound: %q", viper.GetString("vitest_config"))
}

// Check that viper picks up the flag values
if viper.GetString("platform") != "python" {
t.Errorf("expected viper platform to be 'python', got %q", viper.GetString("platform"))
Expand Down
Loading
Loading