[SDTEST-3895] Internal local testdrive analyzer

[anmarchenko]

What

Group local test attempts by module, suite, name, and parameters; report failed, flaky, slow, and broadly covered tests with their coverage context. Surface tracer configuration errors and empty coverage separately from application failures.

Part 4/14 of #128, based on main after #136 merged. Next: #138.

Why

Customers need actionable findings rather than raw traffic or retry-inflated test counts.

E2E testing

This PR exposes a component, not a testdrive command yet. Use Go 1.27.1 (as pinned in go.mod) on this PR’s checkout. Create .qa-intake/main.go under the DDTest root with the following manual harness, then run go run ./.qa-intake. It prints a loopback URL and artifact directory and waits while QA sends traffic.

package main

import (
    "bufio"
    "fmt"
    "os"
    "github.com/DataDog/ddtest/internal/testdrive"
    "github.com/DataDog/ddtest/internal/testdrive/intake"
)

func main() {
    root, err := os.MkdirTemp("", "ddtest-manual-qa-")
    if err != nil { panic(err) }
    session, err := testdrive.NewSession(root)
    if err != nil { panic(err) }
    server, err := intake.Start(session.Directory())
    if err != nil { panic(err) }
    fmt.Println("URL:", server.URL(), "artifacts:", session.Directory())
    fmt.Println("Press Enter after sending traffic to inspect results and close.")
    bufio.NewScanner(os.Stdin).Scan()
    if err := server.Close(); err != nil { panic(err) }
    findings, err := server.Facts()
    fmt.Printf("findings: %+v\nerror: %v\n", findings, err)
}

In a separate temporary directory, prepare a real Node.js 22+ project:

git init -q
npm init -y
npm install --save-dev jest@30.5.1 dd-trace@6.15.0
printf 'module.exports = (a, b) => a + b;\n' > sum.js
printf "const sum = require('./sum'); test('adds', () => expect(sum(1, 2)).toBe(3));\n" > sum.test.js

Set QA_INTAKE_URL to the harness URL and run the tracer against that local intake:

DD_CIVISIBILITY_ENABLED=true DD_CIVISIBILITY_AGENTLESS_ENABLED=true \
DD_API_KEY=manual-qa DD_CIVISIBILITY_AGENTLESS_URL="$QA_INTAKE_URL" \
DD_CIVISIBILITY_GIT_UPLOAD_ENABLED=false DD_INSTRUMENTATION_TELEMETRY_ENABLED=false \
NODE_OPTIONS='-r dd-trace/ci/init' npx jest --runInBand --config '{}'

1. Run the passing adds test, then press Enter in the harness. Expect one logical test even if multiple retry events arrived, a passing final status, and coverage associated with sum.js.

2. In a fresh harness session, change the expectation to toBe(4) and rerun Jest. Expect Jest to fail and the printed findings to list adds as failed with its assertion error. Received instrumentation must still be visible.

3. In a fresh session, use a module-level attempt counter so a test fails on its first invocation and passes on retry. Confirm in the captured traffic that both outcomes arrived. Expect one logical test with multiple attempts and a flaky finding, not two independent tests. Compare its final status with the tracer’s final-status field.

4. Add three tests that finish immediately and one that waits 500 ms. Run in a fresh session. Expect the slow test in SlowTests and the fast tests absent from that list.

5. In a fresh disposable Jest project, create four separate test files (one test per file): three narrow suites that each load one source module, and one broad suite that loads twelve distinct source modules. Run with coverage in a fresh intake session. Jest emits suite-level coverage, so inspect BroadCoverage for the broad suite, with exactly its twelve source modules plus its test file (13 files). Compare the file list with captured coverage. Putting all four tests in one file gives them shared suite coverage and correctly produces no broad outlier. Verify unrelated coverage IDs with the deterministic HTTP scenario rather than adding unrelated files to a Jest suite.

6. To verify analysis when coverage is absent, create the deterministic harness below and run go run ./.qa-findings no-coverage from the DDTest root. It sends test events without a coverage request. Expect 6 events / 5 logical tests, zero covered tests, zero empty entries, no broad-coverage findings, and unchanged failed/flaky/slow classifications. With the pinned dd-trace 6.15.0 setup, DD_CIVISIBILITY_CODE_COVERAGE_ENABLED=false and DD_CIVISIBILITY_ITR_ENABLED=false do not disable collection: the tracer uses the intake settings response, which enables code_coverage. This deterministic scenario verifies absent-coverage analysis; it does not certify a real-Jest coverage-disable flag.

7. In a fresh harness, send a MessagePack test-cycle payload containing a passing test and a test_session_end event whose content.meta includes _dd.ci.library_configuration_error.skippable_tests: "true". Use Python in the disposable QA environment (python -m pip install msgpack) and the snippet below. Close the harness: expect one test, no failed tests, and ConfigurationErrors:[skippable_tests]. Repeat with the same true tag on a suite event: it must appear only once. Change it to "false": it must disappear.

import os, urllib.request, msgpack
payload = {"events": [
    {"type": "test", "content": {"meta": {"test.name": "passes", "test.status": "pass"}}},
    {"type": "test_session_end", "content": {"meta": {"_dd.ci.library_configuration_error.skippable_tests": "true"}}}
]}
request = urllib.request.Request(os.environ["QA_INTAKE_URL"] + "/api/v2/citestcycle", data=msgpack.packb(payload), headers={"Content-Type": "application/msgpack"})
print(urllib.request.urlopen(request).status)

Cleanup: stop any remaining harness processes and remove .qa-intake plus the printed temporary QA directories after inspecting the artifacts.
Also remove the temporary Jest repository.

Additional empty-coverage scenario: use the deterministic fixtures from #136’s E2E plan against this PR’s intake harness. POST the valid events and coverage-empty.msgpack, then inspect the printed findings. Expect EmptyCoverageEntryCount: 1, unchanged test/event counts, and zero covered tests. Repeat with mixed valid/empty coverage and empty suite coverage: both must retain that diagnostic and contribute no coverage. A valid-only control must have zero empty entries and retain its coverage association. Remove the temporary fixtures and harness after inspection.

For deterministic identity and coverage regression checks, create .qa-findings/main.go under the repository root with the following harness (Go 1.27.1, no external service or credentials required):

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"mime/multipart"
	"net/http"
	"net/textproto"
	"os"
	"time"

	"github.com/DataDog/ddtest/internal/constants"
	"github.com/DataDog/ddtest/internal/testdrive/intake"
	"github.com/tinylib/msgp/msgp"
)

func main() {
	mode := os.Args[1]
	root, err := os.MkdirTemp("", "ddtest-findings-qa-")
	must(err)
	fmt.Println("Artifacts:", root)
	server, err := intake.Start(root)
	must(err)
	var events, coverages []any
	for i := range 6 {
		meta := map[string]any{"test.module": "module-one", "test.suite": "suite", "test.name": "same", "test.parameters": `{"arguments":{"x":1}}`, "test.status": "pass"}
		duration := time.Millisecond
		switch i {
		case 1, 2:
			meta["test.parameters"] = `{"arguments":{"x":2}}`
			if i == 1 {
				meta["test.status"] = "fail"
			} else {
				meta["test.is_retry"] = "true"
			}
		case 3:
			meta["test.module"] = "module-two"
			meta["test.status"] = "fail"
		case 4:
			meta["test.name"] = "fast"
		case 5:
			meta = map[string]any{"test.status": "pass"}
			duration = time.Second
		}
		events = append(events, map[string]any{"type": "test", "content": map[string]any{
			"test_session_id": 1, "test_suite_id": 2, "span_id": i + 1, "duration": int64(duration), "meta": meta,
		}})
		coverages = append(coverages, map[string]any{"test_session_id": 1, "test_suite_id": 2, "span_id": i + 1, "files": []any{map[string]any{"filename": fmt.Sprintf("source-%d.js", i+1)}}})
	}
	if mode == "single-test" || mode == "single-suite" {
		events = events[:1]
		coverages = coverages[:1]
	}
	if mode == "suite" || mode == "single-suite" {
		coverages = []any{map[string]any{"test_session_id": 1, "test_suite_id": 2, "files": []any{map[string]any{"filename": "shared.js"}}}}
	}
	if mode == "empty" {
		coverages = append(coverages, map[string]any{"span_id": 99, "files": []any{}})
	}
	events = append(events, map[string]any{"type": "test_session_end", "content": map[string]any{"meta": map[string]any{"_dd.ci.library_configuration_error.skippable_tests": "true"}}})
	body, err := msgp.AppendIntf(nil, map[string]any{"events": events})
	must(err)
	post(server.URL()+constants.TestCycleURLPath, "application/msgpack", body)
	if mode != "no-coverage" {
		body, err = msgp.AppendIntf(nil, map[string]any{"coverages": coverages})
		must(err)
		var buf bytes.Buffer
		writer := multipart.NewWriter(&buf)
		part, err := writer.CreatePart(textproto.MIMEHeader{"Content-Disposition": {`form-data; name="coverage"; filename="coverage.msgpack"`}, "Content-Type": {"application/msgpack"}})
		must(err)
		_, err = part.Write(body)
		must(err)
		must(writer.Close())
		post(server.URL()+constants.TestCoverageURLPath, writer.FormDataContentType(), buf.Bytes())
	}
	must(server.Close())
	findings, err := server.Facts()
	must(err)
	output, err := json.MarshalIndent(findings, "", "  ")
	must(err)
	fmt.Println(string(output))
}
func must(err error) {
	if err != nil {
		panic(err)
	}
}
func post(url, contentType string, body []byte) {
	response, err := http.Post(url, contentType, bytes.NewReader(body))
	must(err)
	defer response.Body.Close()
	_, err = io.Copy(io.Discard, response.Body)
	must(err)
	if response.StatusCode != http.StatusOK {
		panic(response.Status)
	}
	fmt.Println("POST:", response.Status)
}

Run each command from the repository root and inspect the JSON:

1. go run ./.qa-findings test: expect HTTP 200 for both requests, 6 test events, 5 logical/covered tests, one failed test from module-two, one flaky parameter-2 test with two attempts, and one slow unnamed test. Every categorized finding must retain the same coverage as its Tests entry. The parameter-1 test must cover only source-1.js; parameter-2 must cover source-2.js and source-3.js. ConfigurationErrors must contain skippable_tests without increasing failed-test counts.
2. go run ./.qa-findings suite: expect the same test counts and categories, each with suite coverage for shared.js, including the unnamed test.
3. go run ./.qa-findings single-test and go run ./.qa-findings single-suite: expect 1 logical/covered test, CoveredFilesMedian 1, and no broad-coverage finding.
4. go run ./.qa-findings empty: expect HTTP 200, 6 events/5 logical tests, EmptyCoverageEntryCount 1, zero covered tests and no coverage on any finding. The entire mixed payload is excluded.
5. go run ./.qa-findings no-coverage: expect 6 events/5 tests, zero covered tests, zero empty entries, and the same failure/flaky/slow classification.

Inspect each printed artifact directory before deleting it. Remove .qa-findings and the printed directories afterward.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-22T11:19:59.735267Z	64ed4ff	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[datadog-official]

 

✅ All CI checks and tests passed. Datadog automation helped this PR pass.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 1 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 95.82%
• Overall Coverage: 91.77% (+0.14%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d323740 | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 64ed4ff4a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

Two critical correctness bugs stand out: the grouping key does not include all test-identity fields (module, parameters, source position), causing distinct tests to be merged as retries and producing false flaky results and wrong counts; and coverage data is propagated only to the Tests slice, leaving FailedTests, FlakyTests, and SlowTests with empty coverage fields. A separate median-calculation defect causes CoveredFilesMedian to be reported as zero whenever exactly one coverage record exists.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 64ed4ff · @DataDog review to ask questions}

[datadog-official]

Two of the most critical correctness issues are: the test identity key omits module and parameter data, causing distinct parameterized or multi-module tests to be collapsed into one (producing false flaky results and wrong counts), and coverage data is attached to the main Tests list only after the FailedTests/FlakyTests/SlowTests lists are already copied, leaving those derived lists without coverage. Additional bugs include unnamed tests losing coverage due to a key mismatch, a zero-median miscalculation for single-record coverage runs, and a quadratic scan when attaching suite-level coverage.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 8d719f3 · @DataDog review to ask questions}

[anmarchenko]

Addressed the seven automated review threads in ab27291 and the four human naming comments in d323740.

• Test identity follows the tracer definitions inspected in Shepherd: module, suite, name and parameters. Source location remains metadata; it does not split retries.
• Coverage is attached before failed/flaky/slow entries are copied, including unnamed tests. Single-record coverage medians are retained, and suite coverage no longer scans every test for every finding.
• Public analysis types are now Test, TestRun, CoverageFact and Facts, with Server.Facts().

On d323740, Go 1.27.1 make test, make lint, and go test -race ./internal/testdrive/... passed. The updated runnable manual harness passed all six live HTTP scenarios: test coverage, suite coverage, both singleton modes, mixed empty coverage, and absent coverage. Empty payload diagnostics remain intact.

GitHub reports all 45 checks successful on this exact head, including the instrumented CI test job and coverage gate: https://github.com/DataDog/ddtest/actions/runs/35878198628.

The dependent PRs #138–#147 have been rebased with explicit leases and their renamed API callers updated. Every resulting head passed full tests, lint and testdrive race checks locally; tree comparisons verified that downstream changes outside intake are only the required API substitutions. Remote CI is now green on all ten exact published descendant heads.

No unresolved review threads remain. This PR is still open and requires human approval before merge; I will continue monitoring reviews, QA and CI.

[anmarchenko]

@autotest review

[anmarchenko]

Independent E2E QA rechecked head d323740 and reported no analyzer defect in the exercised scenarios.

The QA run passed make test, all six deterministic harness modes, and five supplemental HTTP scenarios. Real Node 22.14.0 / Jest 30.5.1 / dd-trace 6.15.0 runs confirmed passing, expected assertion failure, fail-to-pass flakiness, slow-test classification, and broad suite coverage against raw captured traffic.

Corrected two instructions in the E2E plan:

• Broad Jest coverage now explicitly uses four separate test files. Jest emits suite coverage; the broad suite covers its twelve source modules plus its test file. Tests in one file share coverage and correctly produce no outlier.
• The absent-coverage scenario now uses the verified deterministic no-coverage harness. Neither DD_CIVISIBILITY_CODE_COVERAGE_ENABLED=false nor DD_CIVISIBILITY_ITR_ENABLED=false disabled coverage in the pinned tracer setup, which uses the intake's enabled remote code_coverage setting. No real-Jest no-coverage pass is claimed.

This update changes only the PR's manual verification instructions; the code head is unchanged. QA used local intake traffic, with no production backend/UI validation. Temporary QA artifacts and debug logs were subsequently cleaned up at the user’s request. The final independent retest report is the durable results record.

[datadog-official]

Mixed suite and test coverage produces an inconsistent report. The covered-test total includes suite coverage that the selected test-level analysis ignores.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit d323740 · @DataDog review to ask questions}

[anmarchenko]

E2E retest report: PASS

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)
Environment: Local macOS arm64; Go 1.27.1, Node.js 22.14.0, Jest 30.5.1, dd-trace 6.15.0.
Revision tested: d3237409c4c8a57ec480d3d400a067c0aa8b9cf5 (head rechecked after retest).

Verification	Result
Full Go suite, go test -count=1 ./...	PASS, uncached
Six PR-provided deterministic harness modes	PASS: identity/parameter separation, retry aggregation, categorized findings and coverage retention, single-item medians, mixed empty coverage exclusion, absent coverage
Five supplemental HTTP scenarios	PASS: configuration-tag deduplication and false values, empty-only and empty-suite coverage, unrelated coverage IDs
Real Jest passing/failing tests	PASS: one logical test despite retries; expected failing-process exit 1 and assertion details preserved
Real fail-then-pass retry	PASS: two attempts grouped into one flaky test; final status matches captured test.final_status
Three fast tests and one 500 ms test	PASS: only the slow test classified slow
Corrected broad-coverage scenario in a fresh project with four separate suites	PASS: one broad suite with exactly twelve source modules plus its test file (13 files), matching captured coverage
Corrected deterministic no-coverage scenario	PASS: six events / five tests, zero covered tests, zero empty entries, no broad findings; failed/flaky/slow classifications unchanged

Issues found: No analyzer defect reproduced. Both previously reported E2E-plan issues are resolved by the revised instructions: separate Jest suites establish a meaningful broad-coverage comparison; the deterministic harness verifies absent coverage without relying on ineffective disable flags. No real-Jest coverage-disable flag is certified.

Methodology: Ran the component's HTTP intake harness from an isolated checkout, with fresh intake sessions and real instrumented Jest traffic. Compared analyzer attempt counts, final statuses, and exact coverage file lists against captured HTTP payloads; asserted the deterministic scenarios. No production source changes were made.

Datadog UI/backend verification: Not performed. This PR exposes a local analyzer component, and these runs targeted its loopback intake. This report does not claim production ingestion or UI validation.

Temporary QA artifacts and debug logs are being cleaned up at the user's request; this comment records the final results.