Skip to content

Stabilize e2e suite under parallelism + add coverage tooling - #146

Merged
stevenweaver merged 1 commit into
mainfrom
test/e2e-deflake-parallelism
Jul 19, 2026
Merged

Stabilize e2e suite under parallelism + add coverage tooling#146
stevenweaver merged 1 commit into
mainfrom
test/e2e-deflake-parallelism

Conversation

@stevenweaver

Copy link
Copy Markdown
Member

Summary

The Playwright e2e suite passed test-by-test in isolation but flaked ~7 tests when run in parallel. This PR makes the suite reliable under parallelism and adds coverage tooling.

Root causes (adversarially verified):

  • WASM CPU contention — multiple @slow HyPhy analyses (real WASM, ~37s each) running concurrently on one machine stall past timeouts.
  • Fixed waitForTimeout() races — hard waits insufficient under load.
  • The "shared IndexedDB across workers" theory was disproven — Playwright 1.57 isolates IndexedDB per BrowserContext, so freshStart()'s deleteDatabase() can't cross workers. No src/ changes were needed for the flakes.

Changes

  • playwright.config.js — cap local workers to 4, retries: 1 (safety net only; tests pass at retries:0).
  • package.json — split test:e2e into test:e2e:fast (--grep-invert @slow, parallel) and test:e2e:slow (--grep @slow --workers=1, serial WASM). Total 138 tests preserved (118 fast + 20 slow); @slow still runs under both chromium and mobile-chrome.
  • helpers.js + specs 06/07/10/14/16 — replace fixed waits with web-first assertions (loadDemoFile waits on sequence-info; selectMethod asserts value + .method-description; waitForAnalysisCompletion uses expect().toPass()).
  • 10-branch-selector — fixes a real mobile bug (not a flake): on Pixel 5 the advanced-options panel overlaps the r=3 SVG tree node and intercepts pointer events, so the click times out. scrollIntoViewIfNeeded() + click({ force: true }) after asserting visibility. The old test masked this by silently skipping the click via an if (count > 0) guard.

Coverage tooling

  • monocart-reporter collects V8 coverage over CDP (fixture in e2e/fixtures/coverage.js, env-gated behind E2E_COVERAGE=1). ~72% lines over src/.
  • @vitest/coverage-v8 (pinned to vitest 3.0.4) for unit coverage. coverage-e2e/ gitignored.

Validation

All at --retries=0:

  • 7 originally-flaky target tests: 0 failures across ~436 instances
  • 10-branch-selector (both projects, ×12): 72/72
  • WASM @slow lane (serial, ×6, both projects): 36/36
  • Lane split: 138 total = 118 fast + 20 slow, @slow present on mobile

Notes

  • All 18 specs show as modified because their @playwright/test import was repointed to the coverage fixture; only 06/07/10/14/16 have logic changes.
  • Deferred: suite is slow (~14s/test) because freshStart does two page loads and +page.svelte onMount always boots HyPhy WASM even for non-analysis tests. A single-load freshStart would roughly halve most tests.

The Playwright e2e suite passed in isolation but flaked ~7 tests when run
in parallel. Root causes (adversarially verified): WASM CPU contention from
concurrent @slow HyPhy analyses, and fixed waitForTimeout() races that break
under load. The shared-IndexedDB theory was disproven — Playwright isolates
IndexedDB per BrowserContext.

Changes:
- playwright.config.js: cap local workers to 4, retries:1 (safety net only)
- package.json: split test:e2e into test:e2e:fast (--grep-invert @slow) and
  test:e2e:slow (--grep @slow --workers=1) so real-WASM tests run serially
  while the fast suite stays parallel; total 138 tests preserved, @slow still
  runs under both chromium and mobile-chrome
- helpers.js + specs 06/07/10/14/16: replace fixed waits with web-first
  assertions (loadDemoFile waits sequence-info; selectMethod asserts value +
  .method-description; waitForAnalysisCompletion uses expect().toPass())
- 10-branch-selector: fix a real mobile bug (not a flake) — the advanced-
  options panel overlaps the r=3 SVG node and intercepts pointer events on
  Pixel 5; scrollIntoViewIfNeeded() + force click after asserting visibility.
  The old test masked this by silently skipping the click via an if-guard.

E2E coverage tooling: monocart-reporter collects V8 coverage over CDP
(fixture in e2e/fixtures/coverage.js, env-gated in playwright.config.js);
@vitest/coverage-v8 for unit coverage. coverage-e2e/ gitignored.

Validated at --retries=0: 7 target tests + branch-selector 72/72,
WASM lane 36/36, all green.
@stevenweaver
stevenweaver merged commit dfd0638 into main Jul 19, 2026
2 checks passed
@stevenweaver
stevenweaver deleted the test/e2e-deflake-parallelism branch July 19, 2026 03:07
@stevenweaver
stevenweaver restored the test/e2e-deflake-parallelism branch July 31, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant