CIP-005: auto-repair - #26
Merged
Merged
Conversation
Implements CIP-005. Until now nothing regenerated a lost shard: a degraded
block stayed degraded, and every object trended toward unrecoverable on a
schedule set by node churn. CIP-001 bought c0mpute's cost advantage by
spending the durability margin Storj keeps — RS 10/14 tolerates four losses
where RS 29/80 tolerates fifty-one — so fast repair is what makes that trade
defensible rather than reckless.
`c0mpute-placement::repair`:
- Rendezvous election, so exactly one holder repairs a block with no
coordination (DIP-0011). Rotates by round, so a dead winner does not block
a block forever.
- Flap tolerance: a shard is presumed lost only after grace_probes failures
spread over grace_window. Both conditions matter — the count alone
condemns a peer from a burst of probes seconds apart. Repairing a
rebooting node is how a flap becomes a storm.
- Reconstruct from k, verify against the manifest's block hash before
re-encoding. Repairing from unchecked bytes would launder a corrupt block
into fresh shards that agree with each other and not with the manifest.
- Regenerate only the missing shards; rewriting healthy placements would
multiply the repair traffic CIP-001 says the margin cannot absorb.
- Priority ordering (Critical first, Lost last since it cannot be helped)
and a per-pass cap under storm conditions.
`c0mpute storage repair [hash] [--dry-run] [--now]`, sweeping every object by
default and writing the manifest back so later reads follow the new homes.
Also extends selection with PlacementContext, so replacements are chosen
against the domains the survivors already occupy. Without it a block drifts
into one failure domain one repair at a time, each repair individually legal.
Three problems the tests and testnet found, all the same shape — the catalog
has no liveness signal, and every layer that assumes otherwise fails in a way
that looks like success:
1. Election picks among a block's holders, so an operator running `repair` —
who holds nothing — could never win and every repair deferred. Election
is now a mode: honoured by the daemon, bypassed on explicit request.
2. A peer that just died still looks healthy in the catalog, because
reputation and uptime are periodic measurements. Repair selected it as
the destination for the replacement, "succeeded", and left the block
exactly as degraded.
3. A peer that died in an earlier round is not probed at all, because it
holds none of this block's shards. Repair now carries spare candidates
and fails over. Any subset of a valid selection is valid — the per-domain
cap is a maximum — so skipping a dead candidate cannot break diversity.
32 new tests (14 unit, 18 integration). Verified on a 24-node testnet: three
holders killed and repaired onto fresh peers, then three of the *new* holders
killed and repaired again — six dead across two rounds, past the parity budget
of four, with the object still byte-identical. A paused node inside its grace
window is left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx
ThreatCrush Security Scan8 finding(s) HIGH/CRITICAL: 5 | MEDIUM: 3
Snippets are redacted; ThreatCrush never prints matched credential material. |
ralyodio
added a commit
that referenced
this pull request
Aug 29, 2026
…sh-merges) (#27) * feat(storage): CIP-003 cross-node shard placement Implements CIP-003. Until now every shard landed on one disk, which meant the erasure coding was pure overhead with no durability behind it. Blocks are now spread across n peers chosen for reputation and failure-domain diversity, and read back from whichever k answer first. New crate `c0mpute-placement`: peer — PeerInfo, PeerCatalog, and FailureDomain (ASN, falling back to IP prefix, then Unknown). select — choosing n peers under CIP-001's rules. Pure; no network I/O, because a slow peer lookup must not become a slow write. transport — ShardTransport trait, with HTTP and in-memory implementations. distributed— DistributedStorage, composing the three. Two decisions worth calling out. **HTTP first, libp2p later.** CIP-003 assumed the libp2p protocol had to be rewritten before placement could work. It didn't: CIP-002 already ships shard PUT/GET/HEAD endpoints that verify what they are given, so placement was built against a transport trait with an HTTP implementation over those. Cross-node placement works today on a real testnet, and the streaming libp2p protocol becomes a second implementation of an existing trait rather than a blocker. **Placement fails loudly.** CIP-001's durability figures assume shard hosts fail independently; fourteen shards behind one ISP are one sample wearing fourteen hats, and nothing downstream can detect it. So a write that cannot satisfy the diversity policy is an error naming the constraint, not a warning: failure-domain diversity unsatisfiable: 14 shards at most 2 per domain needs 7 distinct domains, but only 1 are available Greedy selection under a per-domain cap is optimal rather than heuristic — the cap is a partition matroid, so a refusal means no assignment would have worked. Also in this change: - Write acknowledges at k + ceil(parity/2) (12 of 14 for standard), so two slow peers do not fail a write; reads request all n and reconstruct from the first k. - `c0mpute storage peer add|ls|rm|ping`, a peers.json registry, and put/get/ info using the network when peers are configured. - CIP-003 sketched the peer score with a `1/(1+rtt/100)` latency term. That lets a fast flaky peer outrank a slow reliable one, which is the opposite of what CIP-001 says matters. Narrowed to a band that breaks ties without overturning a reputation gap; a test pins it. Three bugs found by running it rather than by tests: - `c0mpute storage get` still used the local read path, so an object placed across the network was unreadable — placement worked and retrieval did not. - anyhow's Display drops the cause chain, so the HTTP layer turned "not enough eligible peers: need 14, found 3" into "placing block 0". The whole point of CIP-003 is failing loudly; six sites now format with `{:#}`. - The CLI panicked on SIGPIPE, so `c0mpute storage ls | head` crashed. 43 new tests (27 unit, 16 integration including a real multi-node HTTP test). Verified on a 16-node testnet driven through the CLI: 14 shards on 14 distinct peers, byte-identical read back, still readable with 4 holders killed, refused with 5, and refused outright on a single-domain network. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx * feat(storage): CIP-005 auto-repair (#26) Implements CIP-005. Until now nothing regenerated a lost shard: a degraded block stayed degraded, and every object trended toward unrecoverable on a schedule set by node churn. CIP-001 bought c0mpute's cost advantage by spending the durability margin Storj keeps — RS 10/14 tolerates four losses where RS 29/80 tolerates fifty-one — so fast repair is what makes that trade defensible rather than reckless. `c0mpute-placement::repair`: - Rendezvous election, so exactly one holder repairs a block with no coordination (DIP-0011). Rotates by round, so a dead winner does not block a block forever. - Flap tolerance: a shard is presumed lost only after grace_probes failures spread over grace_window. Both conditions matter — the count alone condemns a peer from a burst of probes seconds apart. Repairing a rebooting node is how a flap becomes a storm. - Reconstruct from k, verify against the manifest's block hash before re-encoding. Repairing from unchecked bytes would launder a corrupt block into fresh shards that agree with each other and not with the manifest. - Regenerate only the missing shards; rewriting healthy placements would multiply the repair traffic CIP-001 says the margin cannot absorb. - Priority ordering (Critical first, Lost last since it cannot be helped) and a per-pass cap under storm conditions. `c0mpute storage repair [hash] [--dry-run] [--now]`, sweeping every object by default and writing the manifest back so later reads follow the new homes. Also extends selection with PlacementContext, so replacements are chosen against the domains the survivors already occupy. Without it a block drifts into one failure domain one repair at a time, each repair individually legal. Three problems the tests and testnet found, all the same shape — the catalog has no liveness signal, and every layer that assumes otherwise fails in a way that looks like success: 1. Election picks among a block's holders, so an operator running `repair` — who holds nothing — could never win and every repair deferred. Election is now a mode: honoured by the daemon, bypassed on explicit request. 2. A peer that just died still looks healthy in the catalog, because reputation and uptime are periodic measurements. Repair selected it as the destination for the replacement, "succeeded", and left the block exactly as degraded. 3. A peer that died in an earlier round is not probed at all, because it holds none of this block's shards. Repair now carries spare candidates and fails over. Any subset of a valid selection is valid — the per-domain cap is a maximum — so skipping a dead candidate cannot break diversity. 32 new tests (14 unit, 18 integration). Verified on a 24-node testnet: three holders killed and repaired onto fresh peers, then three of the *new* holders killed and repaired again — six dead across two rounds, past the parity budget of four, with the object still byte-identical. A paused node inside its grace window is left alone. Claude-Session: https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements CIP-005. Stacked on #25.
Until now nothing regenerated a lost shard: a degraded block stayed degraded, and every object trended toward unrecoverable on a schedule set by node churn. CIP-001 bought c0mpute's cost advantage by spending the durability margin Storj keeps — RS 10/14 tolerates four losses where RS 29/80 tolerates fifty-one — so fast repair is what makes that trade defensible rather than reckless. This is the piece that was missing.
Verified on a real 24-node testnet
Six holders dead — past the parity budget of four — and the object survives, because repair restored redundancy between the rounds. Without it that block is gone.
What's in
c0mpute-placement::repairgrace_probesfailures spread overgrace_window. Both conditions matter — the count alone condemns a peer from a burst of probes seconds apart. Repairing a rebooting node is how a flap becomes a storm, which is what kills p2p storage networks.k, check against the manifest's block hash, then re-encode. Repairing from unchecked bytes would launder a corrupt block into fresh shards that agree with each other and disagree with the manifest.kshards (the 10x amplification the cost model budgets for).Plus
PlacementContext, so replacements are selected against the domains the survivors already occupy. Without it a block drifts into a single failure domain one repair at a time, each repair individually satisfying the cap.CLI:
c0mpute storage repair [hash] [--dry-run] [--now]— sweeps every object by default, writes the manifest back so later reads follow the new homes.Three problems found by running it — all the same shape
The catalog has no liveness signal, and every layer that assumed otherwise failed in a way that looked like success:
repairholds nothing, so they could never win and every repair deferred forever. Election exists to stop fourteen nodes doing the same job, not to stop anyone doing it — it's now a mode, honoured by the daemon, bypassed on explicit request.Found by the testnet, not the suite; all three now have regression tests.
Testing
32 new tests (14 unit, 18 integration). Workspace green: 29 suites. Clippy clean on the crate.
Deliberately not here
Documented in CIP-005's new "Still outstanding" section:
🤖 Generated with Claude Code
https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx