From 827b9079623eaf43a535eb2f5765501ac0aa39f3 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Wed, 2 Sep 2026 17:50:38 +1000 Subject: [PATCH 1/2] docs(security): record the bounded LZ4 path and the unbounded Arrow one (LAB-2504) Two findings from the LAB-2504 cross-SDK audit of the decompression paths. The default read path is sound and now says so: ByteStorage.retrieve is a pass-through to cachekit-core's extract(), which bounds output at min(512 MiB, 1000 x compressed_len) before decompressing. Worth stating explicitly because the msgpack caps sit on already-decompressed bytes, so a reader can easily assume those caps are the protection when they are in fact downstream of it. ArrowSerializer is the exception and gets a warning. Its read path never touches extract(): deserialize() hands the body to pa.ipc.open_file(...).read_all(), which decompresses zstd with no size or ratio limit. Measured 2,570 bytes -> 64 MiB (26,112:1), a ratio core rejects at 1000:1. Neither existing control covers it: the xxHash3-64 prefix is unkeyed so a backend-write attacker recomputes it, and max_value_size is enforced on serialize only, making it a producer-side quota rather than a check on what comes back off the wire. Documented rather than patched because pyarrow exposes no sound bound -- no read-side size cap, no allocation-limiting pool, and write_table emits a single record batch so per-batch accumulation misses the common case, while a post-read_all() check runs after the allocation it should prevent. The real fix reads each buffer's uncompressed-length prefix from the record-batch Flatbuffers metadata, which is a new parser on untrusted input and wants its own review: LAB-2730. Until then the warning names the mitigations that do work today. --- SECURITY.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/SECURITY.md b/SECURITY.md index 008160a0..1db072f7 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -129,6 +129,21 @@ We use MessagePack (safe binary serialization) with type preservation via schema + import msgpack # Safe binary serialization ``` +### Bounded Decompression (ByteStorage envelopes) + +The default read path is `decrypt → ByteStorage.retrieve (LZ4 + xxHash3) → +MessagePack decode`. This SDK does **not** implement LZ4: `ByteStorage.retrieve` +in `rust/src/python_bindings.rs` is a pass-through to cachekit-core's +`ByteStorage::retrieve` → `StorageEnvelope::extract`, which bounds the +decompressed output at `min(512 MiB, 1000 × compressed_len)` before +decompressing. The envelope's self-declared `original_size` is not trusted, and +the xxHash3-64 checksum is unkeyed so it does not gate a forging attacker. See +[cachekit-core Decompression limits][core-decompress] for the numbers and the +constrained-runtime caveat. + +The MessagePack size caps sit on the already-decompressed bytes, so they are +downstream of this bound — the core bound is what protects them. + ### Zero-Knowledge Encryption When enabled via `@cache.secure`, client-side AES-256-GCM encryption ensures the server never sees plaintext: @@ -302,6 +317,38 @@ Reports are archived in `reports/security/` for compliance and audit trails. ## Known Limitations +### Arrow IPC Decompression Is Unbounded + +> [!WARNING] +> `ArrowSerializer` (`serializer="arrow"`, `@cache.io`) compresses with zstd by +> default and its read path does **not** go through cachekit-core's bounded +> `extract()`. `deserialize()` hands the body to +> `pa.ipc.open_file(...).read_all()`, which decompresses with no size or ratio +> limit. A 2.5 KB envelope expands to 64 MiB (26,112:1) — a ratio cachekit-core +> rejects at 1000:1. + +The `[8-byte xxHash3-64][Arrow IPC]` envelope does not help: the checksum is +unkeyed, so anyone who can write to the backend recomputes it. `max_value_size` +does not help either — it is enforced on the **write** path only +(`cache_handler.py`), so it is a producer-side quota, not a check on what comes +back off the wire. + +**Exposure**: non-secure Arrow caches on a backend an attacker can write to. +Secure (`@cache.secure`) caches decrypt through AES-256-GCM first, so a forged +payload is rejected before it reaches the reader. + +**Why it is not simply capped**: pyarrow exposes no read-side size limit, no +allocation-limiting memory pool, and `write_table` emits a single record batch — +so per-batch accumulation does not bound the common case, and a +post-`read_all()` check happens after the allocation it is meant to prevent. A +sound bound requires reading each buffer's uncompressed-length prefix from the +record-batch Flatbuffers metadata before decompressing. That is tracked +separately (LAB-2730). + +**Mitigations available now**: use `compression=None` for Arrow caches read from +untrusted backends, use `@cache.secure`, or run with an enforced process memory +limit. + ### Cryptographic Security > [!NOTE] @@ -396,6 +443,7 @@ We appreciate responsible disclosure from the security community. Security resea [core-security]: https://github.com/cachekit-io/cachekit-core/blob/main/SECURITY.md [core-supply-chain]: https://github.com/cachekit-io/cachekit-core/blob/main/SECURITY.md#supply-chain-security [core-kani]: https://github.com/cachekit-io/cachekit-core/blob/main/SECURITY.md#kani-verification +[core-decompress]: https://github.com/cachekit-io/cachekit-core/blob/main/SECURITY.md#decompression-limits [rustsec]: https://rustsec.org/ [cwe-502]: https://cwe.mitre.org/data/definitions/502.html [cwe-532]: https://cwe.mitre.org/data/definitions/532.html From 328b0786763b941804a0f66811e48e45535be746 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Wed, 2 Sep 2026 18:07:24 +1000 Subject: [PATCH 2/2] docs(security): fix a false Arrow mitigation and correct the exposure (LAB-2504) Expert-panel findings on the previous commit. The listed mitigation 'use compression=None' was FALSE and actively harmful -- deserialize() never reads self.compression; pa.ipc.open_file decompresses according to the stored stream's own BodyCompression metadata, so an attacker's forged envelope declares zstd regardless of the reader's setting. Anyone who followed that advice would have believed they were protected. Now stated as a non-mitigation, explicitly. Exposure was wrong in both directions. Narrower: @cache.io is the CachekitIO BACKEND preset and does not select Arrow -- that needs an explicit serializer='arrow' plus the [data] extra. Wider: deserialize() also accepts raw ARROW1 bodies with no checksum at all via the legacy integrity-off branch, so an attacker need not recompute the unkeyed xxh3 prefix the old text said they would. 'A sound bound requires a Flatbuffers walk' was wrong. Uncompressed Arrow IPC allocates in proportion to its own length (measured ratio 1.000), so refusing bodies that declare BodyCompression makes len(body) a real pre-decompression bound in three lines. It costs the compression feature, which is a wire-size and L1-footprint call for the owner rather than a drive-by fix -- so the section now says a bound exists and what it costs, instead of implying none does. The Flatbuffers walk is only needed to KEEP compression. Also records the second measured data point (8,714 -> 256 MiB, 30,805:1) and trims the call-chain symbol names, which would rot on any refactor. --- SECURITY.md | 82 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 49 insertions(+), 33 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index 1db072f7..f2a917e0 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -132,17 +132,16 @@ We use MessagePack (safe binary serialization) with type preservation via schema ### Bounded Decompression (ByteStorage envelopes) The default read path is `decrypt → ByteStorage.retrieve (LZ4 + xxHash3) → -MessagePack decode`. This SDK does **not** implement LZ4: `ByteStorage.retrieve` -in `rust/src/python_bindings.rs` is a pass-through to cachekit-core's -`ByteStorage::retrieve` → `StorageEnvelope::extract`, which bounds the -decompressed output at `min(512 MiB, 1000 × compressed_len)` before -decompressing. The envelope's self-declared `original_size` is not trusted, and -the xxHash3-64 checksum is unkeyed so it does not gate a forging attacker. See +MessagePack decode`. This SDK does **not** implement LZ4 — it delegates to +cachekit-core's bounded `extract()`, which caps the decompressed output at +`min(512 MiB, 1000 × compressed_len)` before decompressing rather than trusting +the envelope's self-declared `original_size`. The xxHash3-64 checksum is +unkeyed, so it detects corruption and does not gate a forging attacker. See [cachekit-core Decompression limits][core-decompress] for the numbers and the constrained-runtime caveat. The MessagePack size caps sit on the already-decompressed bytes, so they are -downstream of this bound — the core bound is what protects them. +downstream of that bound. ### Zero-Knowledge Encryption @@ -320,34 +319,51 @@ Reports are archived in `reports/security/` for compliance and audit trails. ### Arrow IPC Decompression Is Unbounded > [!WARNING] -> `ArrowSerializer` (`serializer="arrow"`, `@cache.io`) compresses with zstd by -> default and its read path does **not** go through cachekit-core's bounded -> `extract()`. `deserialize()` hands the body to -> `pa.ipc.open_file(...).read_all()`, which decompresses with no size or ratio -> limit. A 2.5 KB envelope expands to 64 MiB (26,112:1) — a ratio cachekit-core -> rejects at 1000:1. - -The `[8-byte xxHash3-64][Arrow IPC]` envelope does not help: the checksum is -unkeyed, so anyone who can write to the backend recomputes it. `max_value_size` -does not help either — it is enforced on the **write** path only -(`cache_handler.py`), so it is a producer-side quota, not a check on what comes -back off the wire. +> `ArrowSerializer` (`serializer="arrow"`, requires the `[data]` extra) does not +> read through cachekit-core's bounded `extract()`. `deserialize()` hands the +> body to `pa.ipc.open_file(...).read_all()`, which decompresses with no size or +> ratio limit. Measured: a 2,570-byte envelope expands to 64 MiB (26,112:1), and +> 8,714 bytes to 256 MiB (30,805:1) — ratios cachekit-core rejects at 1000:1. +> Tracked in LAB-2730. + +Neither existing control covers it: + +- **The `[8-byte xxHash3-64][Arrow IPC]` prefix is not authentication.** It is + unkeyed, so a backend-write attacker recomputes it — and they need not + bother, because `deserialize()` also accepts raw `ARROW1` bodies with no + checksum at all (the legacy integrity-off branch). +- **`max_value_size` is enforced on the write path only** (`cache_handler.py`), + so it is a producer-side quota, not a check on bytes coming back off the wire. **Exposure**: non-secure Arrow caches on a backend an attacker can write to. -Secure (`@cache.secure`) caches decrypt through AES-256-GCM first, so a forged -payload is rejected before it reaches the reader. - -**Why it is not simply capped**: pyarrow exposes no read-side size limit, no -allocation-limiting memory pool, and `write_table` emits a single record batch — -so per-batch accumulation does not bound the common case, and a -post-`read_all()` check happens after the allocation it is meant to prevent. A -sound bound requires reading each buffer's uncompressed-length prefix from the -record-batch Flatbuffers metadata before decompressing. That is tracked -separately (LAB-2730). - -**Mitigations available now**: use `compression=None` for Arrow caches read from -untrusted backends, use `@cache.secure`, or run with an enforced process memory -limit. +`arrow_compression` defaults to `"zstd"`, so compression is on by default *once +Arrow is selected*; Arrow itself is opt-in. Secure (`@cache.secure`) caches +authenticate via AES-256-GCM before the reader sees anything, so they are not +exposed. + +**A sound bound exists, and it costs the compression feature.** Uncompressed +Arrow IPC allocates in proportion to its own length (measured ratio 1.000), so +refusing bodies that declare `BodyCompression` on read makes `len(body)` a +genuine pre-decompression bound. That requires writing `compression="none"` too, +or every read of our own entries fails — which is a wire-size and L1-footprint +decision, not a drive-by fix. Keeping compression instead means summing each +buffer's uncompressed-length prefix before decompressing; `pa.ipc.read_message` +exposes the first buffer's prefix but not the rest, so that needs a +bounds-checked walk of the record-batch Flatbuffers metadata. LAB-2730 carries +both options. + +Approaches that do **not** work, so nobody re-derives them: pyarrow exposes no +read-side size limit and no allocation-limiting memory pool; accumulating +`batch.nbytes` across `reader.get_batch(i)` is defeated because a forged +envelope declares one batch (our writer chunks to ~8 MiB, an attacker does not); +and a `table.nbytes` check after `read_all()` runs after the allocation it is +meant to prevent. + +**Mitigations available now**: use `@cache.secure` for Arrow caches on +untrusted backends, or run with an enforced process memory limit. Setting +`compression=None` on the serializer does **not** mitigate — `deserialize()` +decompresses according to the stored stream's own metadata and never consults +that setting. ### Cryptographic Security