fix(bench): exclude sub-timer-resolution cases from the comparison

[nulltask]

Closes #202.

PR #201's benchmark comment reported implausible improvements — player-web.GameplayRecorder +2733%, chart.createBeatResolver +1500%, 205 more cases at +90% or higher — for a PR that only changes how scripts are invoked (tsx → node), not any benchmarked function's implementation.

Root cause

Several exports-benchmark cases (closure-allocating ones especially) run in tens to a few hundred nanoseconds per call under CI's timing budget (--time 50 --warmup-time 25) — below what tinybench's timer can resolve (p50Ms == 0). A percent computed from two independent runs of code this cheap is measurement noise, not signal.

Verified this directly: benchmarking the identical built code twice back-to-back (no code change at all) swings a case like chart.eventToBeat by ~58% (60.5M → 24.4M ops/s) purely from run-to-run noise. CI's base/head split (separate git worktrees / processes, see .github/workflows/ci.yml benchmark-pr job) is exactly the setup where that noise reads as a "regression" or "improvement" against the other run.

(I initially thought this meant PR #201 changed nothing real — see the correction comment on #202. It turns out tsx's esbuild-transpiled output and Node's type-stripped source do differ in how V8 optimizes closure-allocating call sites under a short timing budget, so #201 is a real, reproducible speedup for those cases. The percentages are directionally real but the magnitude reported for sub-tick-resolution cases isn't trustworthy either way, which is the actual bug this PR fixes.)

Fix

Added a reliability floor to scripts/bench/compare-results.ts: compareSnapshots now excludes any case whose base or head median per-call latency (p50Ms) is at or below 0.001ms from the regression/improvement/unchanged classification entirely — not folded into "unchanged", since a percent computed from noise isn't a real "no change" either. Excluded cases get their own count (ComparisonSummary.unreliableCaseCount) and a labeled "Excluded (sub-timer-resolution)" section in the markdown report, so they stay visible instead of silently vanishing.

countUnreliableCases is exported separately from compareSnapshots so existing callers of compareSnapshots (and its existing tests) keep returning ComparedRow[] unchanged — no signature break.

Testing

• Added test cases covering: a case excluded via head-side noise, one excluded via base-side noise, one that stays included just above the floor, and countUnreliableCases counting only comparable (both-sides-present) keys.
• Full build/lint/typecheck/test green (1975 tests).
• Manually generated two real benchmark snapshots with CI's exact timing and compared them: 71 of 75 comparable chart/utils cases were correctly excluded as sub-timer-resolution noise, and the two real regressions/one real improvement that remained are outside the floor.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	be-music-player-demo	8915c39	Commit Preview URL Branch Preview URL	Sep 06 2026, 07:41 AM

[github-actions]

Exports Benchmark

• Base SHA: cb4c24a4da0d
• Head SHA: 8915c3969209
• Comparable cases: 83
• Regression threshold: 8.00%
• Overall verdict uses the median change across cases.
• Per-case lists compare median ops/s, not mean.
• Base runs: median of 3
• Head runs: median of 3

Summary

Metric	Value
Overall	unchanged
Median change	+0.00%
Mean change	-0.30%
Cases improved (>= threshold)	0
Cases regressed (<= -threshold)	1
Cases unchanged	82
Cases excluded (sub-timer-resolution)	160
Head benchmarked cases	243
Head skipped cases	21

Top Regressions

API	Base median ops/s	Head median ops/s	Change
player-web.buildAudioBus	698324.02	468164.79	-32.96%

Top Improvements

No improvement over threshold.

Excluded (sub-timer-resolution)

Per-call latency at or below 0.001ms on at least one side — the reported time is measurement noise, not the case's real cost, so no percent change is shown.

API	Base median ops/s	Head median ops/s
audio-renderer.createTimingBuildContext	1923076.93	1953124.99
chart.compareEvents	10416666.69	10416666.69
chart.createBeatResolver	20833333.39	20833333.39
chart.eventToBeat	13888889.10	13888889.10
chart.exWavVolumeCentibelsToLinearGain	15625000.04	15625000.04
chart.getMeasureBeats	20833332.60	20833332.60
chart.isBmsBgmVolumeChangeChannel	17857142.61	17857142.61
chart.isBmsDynamicVolumeChangeChannel	15625000.04	17857142.61
chart.isBmsKeyVolumeChangeChannel	17857142.61	17857142.61
chart.isBmsLongNoteChannel	20833332.60	20833332.60
chart.isLandmineChannel	20833332.60	20833332.60
chart.isPlayableChannel	20833332.60	20833332.60