CIP-003: cross-node shard placement

[ralyodio]

Implements CIP-003. Rebuilt on top of #24, so the stack sits on master — this replaces #23, which was based on feat/cip-002-storage-api, a branch stranded off-master by the #20 squash-merge. Same commit, replanted; no content changes.

The diff is placement only.

Until now every shard landed on one disk, which meant the erasure coding was pure overhead with no durability behind it. Blocks are now spread across n peers chosen for reputation and failure-domain diversity, and read back from whichever k answer first.

Verified on a real 16-node testnet, driven through the CLI

=== distributed put (RS 10/14 across 16 nodes) ===
  placing across 16 peer(s) in 16 failure domain(s)
  1.9 MiB in 1 block(s), 14 shards, tier standard (1.4x expansion, 2.7 MiB raw)

  14 shards on 14 distinct peers
  14 node data directories contain shard files

=== read it back over the network ===
OK byte-identical from 14 remote peers

=== kill 4 of the 14 shard holders, read again ===
OK reconstructed from parity with 4 holders down

=== kill a 5th holder — must fail loudly, not silently ===
correctly refused: need 10 shards, got 9

=== a network with no failure-domain diversity is refused ===
failure-domain diversity unsatisfiable: 14 shards at most 2 per domain
needs 7 distinct domains, but only 1 are available (2 shards could be placed)

New crate: c0mpute-placement

module	what it does
peer	PeerInfo, PeerCatalog, FailureDomain (ASN → IP prefix → Unknown)
select	choosing n peers under CIP-001's rules. Pure — no network I/O, because a slow peer lookup must not become a slow write
transport	ShardTransport trait, with HTTP and in-memory implementations
distributed	DistributedStorage, composing the three

Two decisions worth your attention

HTTP first, libp2p later. CIP-003 assumed the libp2p protocol had to be rewritten before placement could work. It didn't. CIP-002 already ships shard PUT/GET/HEAD endpoints that verify what they're given, so placement was built against a transport trait with an HTTP implementation over those. Cross-node placement works today; the streaming libp2p protocol becomes a second implementation of an existing trait rather than a blocker. It's still worth doing (removes an HTTP hop, gives CIP-005 a batched Have probe) — it's just independently schedulable now. CIP-003 is updated to say so.

Placement fails loudly. CIP-001's durability figures assume shard hosts fail independently. Fourteen shards behind one ISP are one sample wearing fourteen hats, and nothing downstream can detect that the assumption was broken. So a write that can't satisfy the diversity policy is an error naming the constraint, not a warning. --insecure-ignore-diversity exists for testnets and says what it costs.

Greedy selection under a per-domain cap is optimal, not heuristic — the cap is a partition matroid, so greedy is optimal over it. A DiversityUnsatisfiable result means no other assignment would have worked, so there's no backtracking and no better answer being missed.

Also here

• Write acknowledges at k + ceil(parity/2) (12 of 14 for standard), so two slow peers don't fail a write. Reads request all n and reconstruct from the first k — costs 1.4x read bandwidth, buys the k-th fastest peer instead of the slowest of a chosen k.
• c0mpute storage peer add|ls|rm|ping plus a peers.json registry; put/get/info use the network when peers are configured. peer ls reports readiness per tier against the domain count, which is the number that actually decides durability.
• CIP-003's peer score was wrong and the implementation corrected it. The sketched 1/(1+rtt/100) latency term makes a 400 ms peer score 20% below a 1 ms one — enough for a fast flaky node to outrank a slow reliable one, the opposite of what CIP-001 says matters. Narrowed to a band that breaks ties without overturning a reputation gap; a test pins it.

Three bugs found by running it, not by tests

1. c0mpute storage get still used the local read path, so an object placed across the network was unreadable — placement worked and retrieval didn't. Caught by the testnet script, not the suite.
2. anyhow's Display drops the cause chain, so the HTTP layer turned not enough eligible peers: need 14, found 3 into placing block 0. The entire point of CIP-003 is failing loudly; six sites now format with {:#}.
3. The CLI panicked on SIGPIPE — c0mpute storage ls | head crashed with failed printing to stdout: Broken pipe. Restored the default disposition.

Testing

43 new tests (27 unit, 16 integration) including a real multi-node HTTP test that stands up five gateway servers and places across them. Whole workspace green (28 suites); clippy-clean on the new crate.

Not in scope

Repair (CIP-005) — nothing yet regenerates a lost shard, so a degraded block stays degraded; info reports it. Stale host_hint resolution via Kad provider records, and the batched Have probe, land with the libp2p transport. Payouts are CIP-006.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LsQAuvXkmyHTgnvquLHrRx

[github-actions]

ThreatCrush Security Scan

8 finding(s)

HIGH/CRITICAL: 5 | MEDIUM: 3

Severity	Rule	Location
HIGH	sh-remote-script-execution	scripts/dev-setup.sh:25
HIGH	sh-remote-script-execution	scripts/install.sh:159
HIGH	sh-remote-script-execution	scripts/install.sh:184
HIGH	sh-remote-script-execution	scripts/install.sh:277
HIGH	sh-remote-script-execution	scripts/install.sh:294
MEDIUM	js-unescaped-html-sink	apps/web/src/app/blog/[slug]/page.tsx:53
MEDIUM	js-unescaped-html-sink	apps/web/src/app/layout.tsx:89
MEDIUM	sh-eval-expansion	scripts/dev-setup.sh:35

Snippets are redacted; ThreatCrush never prints matched credential material.