test(benchmark): codec vs per-read cost decomposition (stacked on #9)

[dougnukem]

Stacked on #9. Splits fastp's processing time by kind of work: gzip decompression, parsing and per-read QC, gzip compression, and writing. #9 splits it by stage in time. This split decides which optimisations can pay off.

What it adds (benchmark/codec-io.md has the method, how to read it, and the results):

• scripts/io_variants.sh: the same binary on the same reads with codec stages switched off. Inputs are gz, plain, or BGZF; outputs are gz, plain, or none. plain_none is the per-read compute floor. The one-off cost of making the plain and BGZF inputs goes to prep.tsv.
• scripts/codec_baselines.sh: standalone codec MB/s per thread count (ISA-L, libdeflate levels 1/4/6, pigz, rapidgzip, bgzip).
• scripts/threadcpu.py: per-thread CPU sampler, opt-in from bench_full.py via FASTP_BENCH_THREADS_JSON.
• scripts/decompose.py: per-stage wall and CPU differences, the single-thread reader ceiling, busiest threads, and a check that output digests don't differ across I/O modes.
• bench_full.py: digests plain outputs and allows runs with no -o. TSV columns unchanged.

Results (n2d-highmem-48, 4 complete public datasets, -w 16/48, 9 variants × 2 reps; data in benchmark/results/codec-io-2026-09/):

• Per-read work is ~75% of CPU on every dataset. Output compression is 17–25%, input decompression 1–5%. Removing both codecs saves only 13–26% of wall.
• -w 48 is slower than -w 16 on 3 of 4 datasets (atac 200 → 270 s) and costs 25–45% more CPU. The box is 24 cores × 2 SMT threads.
• BGZF input doesn't help. The parse stays on the reader, and converting to BGZF costs ~5.7 µs of CPU per read.
• Output digests are identical across all 72 variant/thread cells.
• Uncapped parallel inflate: rapidgzip 3.4 GB/s at 16 threads (~3.7× ISA-L's CPU per byte); bgzip on BGZF 9.0 GB/s.

Source facts (in codec-io.md):

• Ordinary .gz input is inflated and parsed by one thread per mate.
• Output compression is already parallel.
• The BGZF pool is max(1, (nproc − (-w+4)) / gz_inputs), which is 1 thread at -w ≥ 44 on 48 vCPUs.

Self-review fixes already in:

• threadcpu role grouping (fp-write/fp-bgzf were merged into one role).
• Decompression baselines write to /dev/null; piping through wc capped them at ~2.5 GB/s.

🤖 Generated with Claude Code