CIP-002: storage HTTP API + c0mpute storage CLI - #21
Merged
Conversation
Implements CIP-002: turns the c0mpute-store engine into a usable service and
puts it behind a CLI. Single node — every shard lands on the local disk and
host_hint stays None; cross-node placement is CIP-003.
c0mpute-store
- Tier (hot / standard / critical) carrying (k, parity) and CIP-001 pricing.
hot is RS k=1, which makes parity shards byte-identical copies: real 3-copy
replication with 1x repair amplification.
- Block layer: objects split into blocks, each independently RS-encoded, so
memory is bounded by block size and a range read touches only the blocks it
needs. CIP-007's random-access files depend on this.
- Manifest v2 (version, block_size, tier, blocks[]) with a shim that still
parses v1.
- block_size_for() scales 4 MiB -> 256 MiB with object size, keeping a 1 TiB
object's manifest under a megabyte instead of ~262k blocks.
- put_stream / read_stream / get_range. put is now a wrapper over put_stream
so there is one write path.
- Storage::list() for enumerating what a node holds.
c0mpute-gateway
- /storage/v1/{objects,shards,manifests,status} with Range support, tier
selection, idempotent PUT, single-flight per hash, and a disk budget.
- Commit-then-verify on every write: bytes must hash to the hash the caller
committed to, or 422.
- ed25519 signed-request auth (DIP-0007 shape) bound to method + path + body
hash, with a 5-minute skew window. Writes require it; reads do not, since
the hash is the capability.
c0mpute-cli
- `c0mpute storage put|get|ls|info|verify|rm|status|tiers|serve`, per
plugins/storage/module.toml. Commands that need later CIPs (volume, mount,
provide) are absent rather than stubbed, and --help says which CIP brings
each one.
Two bugs found by running it, not by the tests:
1. Data loss on rollback. A write that fails its hash commitment used to delete
every shard hash it touched. Shards are content-addressed and shared, so
re-uploading an existing object's bytes under a wrong hash produces the same
shard hashes — the rollback deleted the intact object's shards. One
malformed request destroyed real data. ChunkStore::put_new now reports
whether it created a chunk, and rollback only removes what it created.
Regression tests at both the store and HTTP level.
2. Tracing wrote to stdout, so `HASH=$(c0mpute storage put f)` captured log
lines and every scripted use broke. Diagnostics now go to stderr; daemon
mode is unaffected because it points both at the same log file.
Also: Config's api/storage/gateway sections get serde defaults, so a partial
config.toml that sets only [storage] root loads instead of erroring.
65 new tests (31 store, 12 auth, 22 HTTP integration); workspace is green and
clippy-clean on the touched crates. Verified end to end against a running
server: 12 MiB round-trips byte-identical, range reads match dd, and an object
still reconstructs after deleting 4 of 14 shards.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx
ThreatCrush Security Scan8 finding(s) HIGH/CRITICAL: 5 | MEDIUM: 3
Snippets are redacted; ThreatCrush never prints matched credential material. |
This was referenced Aug 29, 2026
ralyodio
added a commit
that referenced
this pull request
Aug 29, 2026
…#24) Implements CIP-002: turns the c0mpute-store engine into a usable service and puts it behind a CLI. Single node — every shard lands on the local disk and host_hint stays None; cross-node placement is CIP-003. c0mpute-store - Tier (hot / standard / critical) carrying (k, parity) and CIP-001 pricing. hot is RS k=1, which makes parity shards byte-identical copies: real 3-copy replication with 1x repair amplification. - Block layer: objects split into blocks, each independently RS-encoded, so memory is bounded by block size and a range read touches only the blocks it needs. CIP-007's random-access files depend on this. - Manifest v2 (version, block_size, tier, blocks[]) with a shim that still parses v1. - block_size_for() scales 4 MiB -> 256 MiB with object size, keeping a 1 TiB object's manifest under a megabyte instead of ~262k blocks. - put_stream / read_stream / get_range. put is now a wrapper over put_stream so there is one write path. - Storage::list() for enumerating what a node holds. c0mpute-gateway - /storage/v1/{objects,shards,manifests,status} with Range support, tier selection, idempotent PUT, single-flight per hash, and a disk budget. - Commit-then-verify on every write: bytes must hash to the hash the caller committed to, or 422. - ed25519 signed-request auth (DIP-0007 shape) bound to method + path + body hash, with a 5-minute skew window. Writes require it; reads do not, since the hash is the capability. c0mpute-cli - `c0mpute storage put|get|ls|info|verify|rm|status|tiers|serve`, per plugins/storage/module.toml. Commands that need later CIPs (volume, mount, provide) are absent rather than stubbed, and --help says which CIP brings each one. Two bugs found by running it, not by the tests: 1. Data loss on rollback. A write that fails its hash commitment used to delete every shard hash it touched. Shards are content-addressed and shared, so re-uploading an existing object's bytes under a wrong hash produces the same shard hashes — the rollback deleted the intact object's shards. One malformed request destroyed real data. ChunkStore::put_new now reports whether it created a chunk, and rollback only removes what it created. Regression tests at both the store and HTTP level. 2. Tracing wrote to stdout, so `HASH=$(c0mpute storage put f)` captured log lines and every scripted use broke. Diagnostics now go to stderr; daemon mode is unaffected because it points both at the same log file. Also: Config's api/storage/gateway sections get serde defaults, so a partial config.toml that sets only [storage] root loads instead of erroring. 65 new tests (31 store, 12 auth, 22 HTTP integration); workspace is green and clippy-clean on the touched crates. Verified end to end against a running server: 12 MiB round-trips byte-identical, range reads match dd, and an object still reconstructs after deleting 4 of 14 shards. Claude-Session: https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements CIP-002: turns the
c0mpute-storeengine into a usable service and puts it behind a CLI.Stacked on #20 (the CIPs) — this PR targets
feat/storage-cips, so its diff is implementation only.Single node: every shard lands on the local disk and
host_hintstaysNone. Cross-node placement is CIP-003.What works
Verified end to end against a running server, not just in tests: 12 MiB round-trips byte-identical over HTTP,
Rangereads matchdd, 422/416/404/507 all behave, and an object still reconstructs after losing 4 shards.c0mpute-store
Tier(hot / standard / critical) carrying(k, parity)and CIP-001 pricing.hotis RS with k=1, which makes the parity shards byte-identical to the data shard — genuine 3-copy replication with 1x repair amplification, which is the whole reason CIP-001 wants a replicated tier.version,block_size,tier,blocks[]) with a shim that still parses v1, so the original tests stay meaningful.block_size_for()scales 4 MiB → 256 MiB with object size. A 1 TiB object gets ~4096 blocks and a sub-megabyte manifest instead of ~262k blocks and tens of MB.put_stream/read_stream/get_range;putis now a thin wrapper overput_streamso there is one write path, not two that drift.c0mpute-gateway
/storage/v1/{objects,shards,manifests,status}— Range support, tier selection, idempotent PUT, single-flight per object hash, disk budget.c0mpute-cli — the sub-feature
c0mpute storage put|get|ls|info|verify|rm|status|tiers|serve, matching thecli = "c0mpute storage"thatplugins/storage/module.tomlhas been declaring all along.Commands that need later CIPs (
volume,mount,provide) are absent rather than stubbed, and--helpnames the CIP that brings each one.--helpnever advertises something that doesn't work.Two bugs found by running it, not by the tests
1. Data loss on rollback. A write that failed its hash commitment deleted every shard hash it had touched. Shards are content-addressed and shared, so re-uploading an existing object's bytes under a wrong committed hash produces identical shard hashes — the rollback deleted the intact object's shards. One malformed request, from anyone who can obtain the content, destroyed it.
ChunkStore::put_newnow reports whether a call created a chunk, and rollback removes only what it created. Regression tests at both the store and HTTP level (the original unit test stored nothing beforehand, so there was nothing for the bad write to destroy — which is exactly why it passed). Worth remembering for CIP-004 and CIP-005, which both delete content-addressed data and will meet the same trap.2. Tracing wrote to stdout, so
HASH=$(c0mpute storage put f)captured log lines and every scripted use broke. Diagnostics now go to stderr; daemon mode is unaffected becausedaemonize_workeralready points both streams at the same log file.Also:
Config'sapi/storage/gatewaysections get serde defaults, so a partialconfig.tomlsetting only[storage] rootloads instead of erroring on the sections you left out.Testing
65 new tests — 31 store, 12 auth, 22 HTTP integration. Whole workspace green (25 suites), clippy-clean on the touched crates.
Note that repo CI runs only security scanners (CodeQL, ThreatCrush, gitleaks, semgrep, Socket, bun audit) — there is no Rust build/test job, so
cargo test --workspaceis worth running locally on review. Adding one might be worth a follow-up.Not in scope
Cross-node placement (CIP-003), repair (CIP-005), payouts (CIP-006), the filesystem and mount (CIP-007 → CIP-009). Until CIP-003, erasure coding on a single node is overhead without durability —
c0mpute storage statussays so in as many words rather than implying the data is safe.🤖 Generated with Claude Code
https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx