fix: land CIP-003 + CIP-005 on master (stranded again by stacked squash-merges) - #27
Merged
Conversation
Implements CIP-003. Until now every shard landed on one disk, which meant the
erasure coding was pure overhead with no durability behind it. Blocks are now
spread across n peers chosen for reputation and failure-domain diversity, and
read back from whichever k answer first.
New crate `c0mpute-placement`:
peer — PeerInfo, PeerCatalog, and FailureDomain (ASN, falling back to
IP prefix, then Unknown).
select — choosing n peers under CIP-001's rules. Pure; no network I/O,
because a slow peer lookup must not become a slow write.
transport — ShardTransport trait, with HTTP and in-memory implementations.
distributed— DistributedStorage, composing the three.
Two decisions worth calling out.
**HTTP first, libp2p later.** CIP-003 assumed the libp2p protocol had to be
rewritten before placement could work. It didn't: CIP-002 already ships shard
PUT/GET/HEAD endpoints that verify what they are given, so placement was built
against a transport trait with an HTTP implementation over those. Cross-node
placement works today on a real testnet, and the streaming libp2p protocol
becomes a second implementation of an existing trait rather than a blocker.
**Placement fails loudly.** CIP-001's durability figures assume shard hosts
fail independently; fourteen shards behind one ISP are one sample wearing
fourteen hats, and nothing downstream can detect it. So a write that cannot
satisfy the diversity policy is an error naming the constraint, not a warning:
failure-domain diversity unsatisfiable: 14 shards at most 2 per domain
needs 7 distinct domains, but only 1 are available
Greedy selection under a per-domain cap is optimal rather than heuristic — the
cap is a partition matroid, so a refusal means no assignment would have worked.
Also in this change:
- Write acknowledges at k + ceil(parity/2) (12 of 14 for standard), so two
slow peers do not fail a write; reads request all n and reconstruct from
the first k.
- `c0mpute storage peer add|ls|rm|ping`, a peers.json registry, and put/get/
info using the network when peers are configured.
- CIP-003 sketched the peer score with a `1/(1+rtt/100)` latency term. That
lets a fast flaky peer outrank a slow reliable one, which is the opposite
of what CIP-001 says matters. Narrowed to a band that breaks ties without
overturning a reputation gap; a test pins it.
Three bugs found by running it rather than by tests:
- `c0mpute storage get` still used the local read path, so an object placed
across the network was unreadable — placement worked and retrieval did not.
- anyhow's Display drops the cause chain, so the HTTP layer turned "not
enough eligible peers: need 14, found 3" into "placing block 0". The whole
point of CIP-003 is failing loudly; six sites now format with `{:#}`.
- The CLI panicked on SIGPIPE, so `c0mpute storage ls | head` crashed.
43 new tests (27 unit, 16 integration including a real multi-node HTTP test).
Verified on a 16-node testnet driven through the CLI: 14 shards on 14 distinct
peers, byte-identical read back, still readable with 4 holders killed, refused
with 5, and refused outright on a single-domain network.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx
Implements CIP-005. Until now nothing regenerated a lost shard: a degraded
block stayed degraded, and every object trended toward unrecoverable on a
schedule set by node churn. CIP-001 bought c0mpute's cost advantage by
spending the durability margin Storj keeps — RS 10/14 tolerates four losses
where RS 29/80 tolerates fifty-one — so fast repair is what makes that trade
defensible rather than reckless.
`c0mpute-placement::repair`:
- Rendezvous election, so exactly one holder repairs a block with no
coordination (DIP-0011). Rotates by round, so a dead winner does not block
a block forever.
- Flap tolerance: a shard is presumed lost only after grace_probes failures
spread over grace_window. Both conditions matter — the count alone
condemns a peer from a burst of probes seconds apart. Repairing a
rebooting node is how a flap becomes a storm.
- Reconstruct from k, verify against the manifest's block hash before
re-encoding. Repairing from unchecked bytes would launder a corrupt block
into fresh shards that agree with each other and not with the manifest.
- Regenerate only the missing shards; rewriting healthy placements would
multiply the repair traffic CIP-001 says the margin cannot absorb.
- Priority ordering (Critical first, Lost last since it cannot be helped)
and a per-pass cap under storm conditions.
`c0mpute storage repair [hash] [--dry-run] [--now]`, sweeping every object by
default and writing the manifest back so later reads follow the new homes.
Also extends selection with PlacementContext, so replacements are chosen
against the domains the survivors already occupy. Without it a block drifts
into one failure domain one repair at a time, each repair individually legal.
Three problems the tests and testnet found, all the same shape — the catalog
has no liveness signal, and every layer that assumes otherwise fails in a way
that looks like success:
1. Election picks among a block's holders, so an operator running `repair` —
who holds nothing — could never win and every repair deferred. Election
is now a mode: honoured by the daemon, bypassed on explicit request.
2. A peer that just died still looks healthy in the catalog, because
reputation and uptime are periodic measurements. Repair selected it as
the destination for the replacement, "succeeded", and left the block
exactly as degraded.
3. A peer that died in an earlier round is not probed at all, because it
holds none of this block's shards. Repair now carries spare candidates
and fails over. Any subset of a valid selection is valid — the per-domain
cap is a maximum — so skipping a dead candidate cannot break diversity.
32 new tests (14 unit, 18 integration). Verified on a 24-node testnet: three
holders killed and repaired onto fresh peers, then three of the *new* holders
killed and repaired again — six dead across two rounds, past the parity budget
of four, with the object still byte-identical. A paused node inside its grace
window is left alone.
Claude-Session: https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ThreatCrush Security Scan8 finding(s) HIGH/CRITICAL: 5 | MEDIUM: 3
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands CIP-003 and CIP-005 on
master. Both were approved and merged (#25, #26) but never reached master — the same squash-merge stranding that #24 fixed for CIP-002.One PR, based directly on
master, no stacking. That is the actual fix: the previous structure could not survive squash-merge, and re-creating it would strand the work a third time.Why it happened again
The three PRs were merged within seven seconds of each other, each into its own base:
masterfix/recover-cip-002feat/cip-003-placement-v2A squash-merge replaces the branch's commits with one new commit on the target. The branch is then behind master and no longer a route to it, so anything merged into it afterwards goes nowhere. Stacked PRs only work here if each is merged and its child retargeted to master before the child merges — which is fragile enough that the right answer is not to stack at all in this repo.
My fault for building the stack; this PR removes it.
Contents
Two cherry-picks onto
master, unchanged from what you reviewed:3dc499c— CIP-003: cross-node shard placement. Newc0mpute-placementcrate: reputation- and failure-domain-aware peer selection,ShardTransport(HTTP over the CIP-002 endpoints, plus in-memory),DistributedStorage,c0mpute storage peer add|ls|rm|ping. Placement fails loudly when diversity can't be satisfied.85c3d3d— CIP-005: auto-repair. Rendezvous election, flap tolerance, verify-before-rebuild, minimal regeneration, diversity-aware replacement with failover,c0mpute storage repair [--dry-run] [--now].Verified on this branch
cargo build --workspaceclean.cargo test --workspace: 29 suites green, including the 72 tests inc0mpute-placement(38 unit, 16 placement integration, 18 repair integration).The Rust CI gate (#22) is on master now, so
build + testruns on this PR — unlike #25/#26, whose bases predated it.After merging
fix/recover-cip-002,feat/cip-003-placement-v2,feat/cip-005-repair,feat/storage-cips,feat/cip-002-storage-apiandfeat/cip-003-placementare all dead ends and safe to delete.🤖 Generated with Claude Code
https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx