Test output: buffer the result stream and drop the per-line flush (~6x faster file output)

[antalvdb]

Summary

Writing test results is dominated by write() syscalls. Profiling a batch test run shows ~82% of the time in __write_nocancel: the default std::ofstream buffer is tiny, so emitting one result per test instance becomes a flood of small writes, and show_results() additionally flushes the stream on every line via std::endl.

This:

• gives outStream a 1 MB buffer (a per-experiment member, set with rdbuf()->pubsetbuf() before open()), so lines batch into a handful of large writes, and
• writes '\n' instead of std::endl in show_results() so the per-line flush doesn't defeat the buffer.

The stream is flushed when it is closed at the end of testing, so the output is unchanged.

Measured impact

IGTree, 512k test instances written to a file (reused saved instance base):

	wall time	throughput	instructions
before	~31 s	~17k inst/s	~430 B
after	~4.6 s	~111k inst/s	~36 B

≈ 6.5× faster wall-clock, ≈ 12× fewer instructions. Both changes matter: the buffer alone takes ~430 B → ~47 B (even with endl), and '\n' then takes ~47 B → ~36 B; either change alone with the default buffer makes no difference. As a side effect this shows the actual IGTree classification is only ~17 B instructions for 512k (~33k each) — the rest was write-syscall overhead.

Correctness

Output is byte-identical and complete — verified on IGTree (512k lines) and TRIBL2 (20k), both plain and with +v db (distributions printed).

Notes

• pubsetbuf must precede open() to take effect; this is honoured by both libc++ and libstdc++.
• The rarer neighbour-output path (NS_Test) still uses endl; only the common show_results() path is changed here.

Posting for your consideration — happy to adjust (e.g. buffer size, or making it configurable).

🤖 Generated with Claude Code

[kosloot]

merged into the develop branch