feat(compressors): add streaming compression crate

[martintmk]

Adds compressors, a streaming compression crate for bytesbuf byte sequences.

Five formats, each behind a cargo feature of its own: deflate, zlib, gzip, brotli and zstd. None is enabled by default, so a build that speaks only brotli never compiles flate2.

let data = BytesView::copy_from_slice("hello", memory);
let compressed = gzip::compress(data, Resources::global())?;
assert_eq!(gzip::decompress(compressed, Resources::global())?.to_vec(), b"hello".to_vec());

Native bytesbuf integration

Input is read segment by segment straight out of a BytesView, and output is written into the uninitialized spare capacity of a BytesBuf. A view is a chain of segments, so nothing is flattened into a contiguous buffer on the way in and nothing is copied out of a scratch buffer on the way back. Every allocation comes from the caller's own memory provider.

Resource pooling

Resources carries what a codec draws on -- a memory provider and recycled engine state -- and is what every API takes instead of the two separately.

Building a compressor allocates and initializes a substantial amount of state; on a small message that setup can cost as much as the compression itself. Recycling it is therefore on by default, so a service compressing many small bodies spends its budget compressing rather than getting ready to. Resources::global() shares one set process-wide, enable_pooling(n) sizes or disables it, and recycling is transparent: it applies to the engines that benefit and quietly skips the rest.

Streamed compression and decompression

A codec is a state machine, not a one-shot transform, so a stream of any length moves through it with a bounded working set -- one pending input view and one output chunk, however many gigabytes pass through. Behind the futures-stream feature, CompressionStream presents that as a futures_core::Stream, turning any stream of byte sequences into its compressed or decompressed counterpart.

Runtime format selection

Format resolves a Content-Encoding token when the format is a peer's decision rather than yours, and CompressorBuilder::build_format produces an operation for it when the level or the chunk size matters.

Bounded decompression

Every one of these formats can expand its input by orders of magnitude. Nothing in the crate accumulates, so the exposure is in what a caller buffers: DecompressorLimits documents what each format bounds by default, why a ratio alone is not protection, and what to set for untrusted input.

Shape of the API

• CompressorBuilder<T> / DecompressorBuilder<T> carry every setting that means the same thing in every format. The type parameter names the format: <()> has not chosen one and gains a build_gzip-style method per enabled format plus build_format(Format, ..); <Brotli> gains brotli's quality, window and content mode.
• Brotli and zstd validate their configuration as they apply it, so their build returns a BuildError rather than deferring the failure to the first chunk.
• compress / decompress at the crate root take any operation, statically dispatched.
• core::Compression is the contract the formats share, so an API can name an operation: impl Compression<Mode = Compress> accepts any compressor and no decompressor.

Testing

• One contract suite runs every format through the same scenarios, so a format that behaves differently from its siblings fails there rather than surprising a consumer.
• 100% line coverage, no surviving mutants.
• Clippy clean across the feature matrix, including a build with no format at all.

[github-actions]

✅ Version increments look sufficient

cargo semver-checks compared the 1 crate(s) this PR publishes against their previous version-bump commit in git history. Every version increment is sufficient for the detected API changes.

Crate	Baseline	Baseline commit	This PR	Minimum required	Status
compressors	new crate	—	0.1.0	0.1.0	✅ ok

This check is informational and does not block the merge.

View the check run

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.0%. Comparing base (5899abe) to head (05ec5ee).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##             main     #722     +/-   ##
=========================================
  Coverage   100.0%   100.0%             
=========================================
  Files         584      609     +25     
  Lines       62909    66105   +3196     
=========================================
+ Hits        62909    66105   +3196     

Flag	Coverage Δ
linux	75.8% <100.0%> (?)
linux-arm	75.8% <100.0%> (?)
windows	80.1% <100.0%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[martintmk]

can you run these benchmarks and report the time/allocations results?

[martintmk]

🤖 On it - running the full benchmark suite now and will report time and allocation figures.

---

_{AI response generated by rocket}

[martintmk]

🤖 Ran the full suite on Windows (x86_64, 8 cores). Criterion medians; allocation figures from alloc_tracker, which counts allocations made through Rust's global allocator.

Read the zstd rows with care. zstd allocates its contexts through its own allocator rather than Rust's, so its allocation counts understate the true cost. Its timings are unaffected.

Whole-buffer throughput, by payload size

Format	1 KiB	64 KiB	1 MiB
deflate	15.4 µs	618 µs	9.85 ms
zlib	16.1 µs	741 µs	9.20 ms
gzip	21.6 µs	585 µs	11.0 ms
brotli	741 µs	5.47 ms	36.6 ms
zstd	9.87 µs	152 µs	2.12 ms

Decompression:

Format	1 KiB	64 KiB	1 MiB
deflate	4.64 µs	40.0 µs	906 µs
zlib	4.81 µs	43.8 µs	962 µs
gzip	7.21 µs	120 µs	1.96 ms
brotli	12.5 µs	118 µs	2.10 ms
zstd	4.66 µs	46.0 µs	634 µs

Allocations per operation

The claim this crate exists for: input is never flattened and output is written into caller memory, so allocation count is flat in the payload size.

Operation	1 KiB	64 KiB	1 MiB
deflate / zlib / gzip compress	2 allocs, 1 080 B	2 allocs, 1 080 B	3 allocs, 1 598 B
deflate / zlib decompress	2 allocs, 1 120 B	2 allocs, 1 120 B	4 allocs, 2 144 B
gzip decompress	3 allocs, 48 672 B	3 allocs, 48 672 B	5 allocs, 49 696 B
zstd compress	2 allocs, 968 B	2 allocs, 968 B	3 allocs, 1 480 B
zstd decompress	2 allocs, 1 008 B	2 allocs, 1 008 B	4 allocs, 2 032 B
brotli compress	23 allocs	23 allocs	40 allocs
brotli decompress	15 allocs	15 allocs	17 allocs

Compressing a megabyte through the deflate family costs three allocations and ~1.6 KB. Brotli's engine allocates its own workspace, which is why its counts are an order of magnitude higher and largely independent of this crate.

What pooling buys (4 KiB payload)

Format	Compress fresh → pooled	Bytes fresh → pooled
deflate	37.3 µs → 30.2 µs (-19%)	381 112 B → 1 080 B (-99.7%)
zlib	33.8 µs → 28.0 µs (-17%)	381 112 B → 1 080 B
gzip	44.0 µs → 31.6 µs (-28%)	381 112 B → 1 080 B
zstd	24.8 µs → 14.8 µs (-40%)	(allocates outside Rust)
brotli	1.351 ms → 1.339 ms (-1%)	unchanged -- brotli exposes no reset, so it is not pooled

Decompression: deflate 7.54 → 6.72 µs, zlib 8.42 → 6.52 µs, zstd 7.76 → 7.11 µs. Gzip decompressors are deliberately not pooled (the engine's reset cannot express gzip framing), and the numbers show it: 11.5 → 12.7 µs, i.e. noise.

The 353 KB saved per pooled deflate-family compressor is the headline. That is the engine's window and hash tables, allocated and thrown away per message without pooling.

Segmentation -- the bytesbuf claim

Compressing the same 1 MiB payload delivered as different span layouts:

Layout	Time	Allocations
contiguous	478 µs	2
16 KiB spans	484 µs	2
1 KiB spans	604 µs	3
64 B spans	1.88 ms	3

A view split into 16 KiB spans costs 1.3% over contiguous, with the same two allocations -- no flattening happens. Even a pathological 64-byte-span view stays at three allocations; only per-segment call overhead grows.

Output chunk size

Chunk	Time	Allocations
1 KiB	2.23 ms	6
8 KiB	2.18 ms	2
64 KiB (default)	2.79 ms	2
512 KiB	2.18 ms	2

Flat from 8 KiB up, so the default is comfortably inside the plateau.

Levels

Format	L1	L6	L9	Ratio at L1 → L9
deflate	94.4 µs	489 µs	938 µs	4.40 → 7.51
gzip	171 µs	564 µs	1.00 ms	4.39 → 7.49
brotli	285 µs	4.78 ms	104.5 ms	7.02 → 10.10
zstd	97.0 µs	108 µs	2.36 ms	8.68 → 9.20

Brotli level 9 costs 367x level 1 for 44% better compression; zstd reaches a better ratio than brotli level 1 at a quarter of brotli's level-6 cost. This is why the portable Level scale is anchored on each format's balanced point rather than stretched across its native range.

Brotli window

Window	Compress	Decompress	Ratio
2^10	779 µs	11.0 µs	8.75
2^16	772 µs	11.2 µs	10.26
2^18	562 µs	11.1 µs	9.61
2^22 (default)	618 µs	11.4 µs	9.61

Confirms what WindowSize documents: the ratio is not monotonic in the window, a small window is not reliably cheaper, and decompressor cost tracks the data rather than the declared window. Leave it alone unless a measurement on real payloads says otherwise.

---

_{AI response generated by rocket}

[martintmk]

we don't need this API for now

[martintmk]

🔄 [AspBot] ## Automated multi-facet review — PR #722 (feat(compressors): add streaming compression crate)

This PR was reviewed across build, correctness, complexity, consolidation, idiomaticity, documentation, security, and performance facets. Overall this is a well-engineered, defensively-written, and unusually well-documented crate. The typestate builders, sealed traits, canonical error type, MaybeUninit-based zero-copy output path, and saturating limit arithmetic are all high quality.

Overall assessment: REQUEST CHANGES — one High-severity, safe-by-default hardening item; everything else is Medium/Low polish.

---

🔴 High

H1 — Decompression is effectively UNBOUNDED BY DEFAULT for every format (CWE-409/770/400)
FormatLimits::new(max_ratio, max_output) has no stream-count parameter, and per-format decompressor defaults ship no absolute output cap — brotli defaults to new(None, None) (zero bounds), zstd to ratio-only 250,000×, deflate/gzip to ratio-only 1,100×. decompress() accumulates all output into one BytesBuf, so a small crafted body can expand to multi-GB → OOM DoS on the crate's stated use case (decompressing untrusted Content-Encoding bodies). Ratio-only caps are not real protection (250,000× lets ~1 MiB → ~244 GiB). The enforcement mechanism itself is sound — only the default policy is permissive.

• Fix (prepared): add a max_streams param to FormatLimits::new and ship conservative safe-by-default caps for every decompressor — 64 MiB absolute output + 1024 streams — with explicit opt-out via with_max_output_len / with_max_streams / UNLIMITED. Values chosen to sit above the largest legitimate default-path test payload (~22.5 MiB) and concatenation count (2–3), so existing tests remain green. Also update the two doc lines that now contradict the bounded defaults (limits.rs:67-68, flate/mod.rs:21).

🟠 Medium

• Security M1 — zstd decompressor window-log defaults to 128 MiB for untrusted input (zstd/codec.rs:190-194); consider a stricter max_window_log default or document the per-stream cost.
• Security/Correctness M2/M3 — gzip multi-member: unbounded member count + a fresh new_gzip inflate-state allocation per member (raw/zlib recycle via reset(), only gzip reallocates) → ~1000× alloc/CPU amplification from many tiny empty members. Truncated subsequent member is mislabeled corrupt_data vs unexpected_end_of_stream. Fix via the default max_streams above + reuse inflate state + EOF label.
• Security M4 — unsafe { output.advance(produced) } (engine.rs:348) is OOB-sound, but delegates the initialized invariant to the safe Codec::step; mark Codec::step unsafe with a # Safety clause or zero-fill in the driver.
• Perf H1 (Medium impact) — brotli & zstd zero-fill the whole output chunk (up to 64 KiB) before every engine step even though both backends are write-only; flate proves it's avoidable via *_uninit. The MaybeUninit abstraction is defeated on the hottest path.
• Perf H2/H3 — empty source chunks enter the full hot codec path (reserve + zero-fill + native step + self-wake); whole-buffer compress/decompress pays streaming-chunking overhead. Perf H4 — the flagship CompressionStream incremental path has zero benchmark coverage.
• Complexity — Pump::pull is a single ~185-line function and the StreamEnd match has 12 guard-heavy arms recomputing max_streams() up to 3×/iteration; extract cohesive helpers.
• Consolidation — the decompressor limit-delegation trio and stream_ended() logic are byte-identical across all three codecs; the unsafe initialize() helper is copy-pasted verbatim in two files (duplicated unsafe is the riskiest kind).

🟡 Low (defense-in-depth / polish)

• stream.rs source is not fused (possible re-poll-after-None panic on caller-supplied Compression); self-wake spin can peg CPU on a perpetually-ready empty source.
• Pooled engine buffers are not zeroized between uses (in-memory remanence; reset-on-checkout already prevents any functional cross-message leak).
• zstd is the only C parser on the untrusted path — add a cargo audit/cargo deny CI gate.
• output_chunk_size has no upper bound; dead done_reported field; brotli collapses NeedsMoreInput/NeedsMoreOutput into one state; crate-root module named core forces ::core:: disambiguation and inconsistent Result spellings; a handful of near-duplicate per-format unit tests could share a harness.

✅ Verified clean

No memory-safety bug, no exploitable panic, no integer-overflow bug. All 4 unsafe blocks proven sound (the task-flagged flate/codec.rs:233 from_raw_parts is test-only and sound). FFI return codes checked; no content-size trust / no huge-alloc bomb pre-allocation; saturating integer math; no PII in error messages; fail-closed config validation; zero-copy input/output paths; no O(n²) append; dependencies current and advisory-clean (flate2 → zlib-rs only, avoiding the C-zlib CVE class). Idiomaticity is exemplary.

Build/clippy/test status

⚠️ Not verified by compiler. The review environment's egress to index.crates.io / static.crates.io was blocked, so cargo build/clippy/test could not fetch dependencies. Source was reviewed at head 17f82b1 via the GitHub API; the H1 fix was implemented on the real tree and parse-checked with rustfmt (clean), and verified complete/non-regressing by static inspection (all 12 FormatLimits::new sites accounted for, enforcement plumbing confirmed end-to-end, existing test payloads confirmed under the new caps). Please run cargo build/test/clippy -p compressors in an environment with crates.io access to confirm.

---

Review performed by an automated multi-agent review team. Line numbers reference head 17f82b1.

[martintmk]

I believe the main highlight of this crate is direct bytesbuf itnegration, resource pooling, and unified compresison API that supports treaming of large data. Metions of "Compression engines normally speak std::io::Read and std::io::Write, which assume a single" should be only secondary highlight.