Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ members = [
"node/crates/c0mpute-core",
"node/crates/c0mpute-net",
"node/crates/c0mpute-store",
"node/crates/c0mpute-placement",
"node/crates/c0mpute-gateway",
"node/crates/c0mpute-verify",
"node/crates/c0mpute-update",
Expand Down Expand Up @@ -65,6 +66,7 @@ rpassword = "7"

c0mpute-proto = { path = "node/crates/c0mpute-proto" }
c0mpute-store = { path = "node/crates/c0mpute-store" }
c0mpute-placement = { path = "node/crates/c0mpute-placement" }
c0mpute-net = { path = "node/crates/c0mpute-net" }
c0mpute-gateway = { path = "node/crates/c0mpute-gateway" }
c0mpute-verify = { path = "node/crates/c0mpute-verify" }
Expand Down
40 changes: 35 additions & 5 deletions docs/prds/003-shard-placement-transport.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
---
cip: 003
title: "Cross-node shard placement and streaming transport"
status: Draft
status: In progress
authors:
- anthony@profullstack.com
created: 2026-08-29
updated: 2026-08-29
implements: DIP-0012 (0012-storage-plugin.md) Phase 3
depends-on: 002
blocks: 005, 006
implementation:
implementation: PR #23 (c0mpute-placement crate; HTTP transport over the CIP-002 shard endpoints; `c0mpute storage peer`)
estimate: "3–4 weeks"
---

Expand Down Expand Up @@ -57,6 +57,20 @@ first thing this CIP fixes.

## Design

### What shipped first: HTTP, not libp2p

The plan below rewrites the libp2p protocol before placement can work. That
ordering turned out to be unnecessary. CIP-002 already ships shard `PUT`/`GET`/
`HEAD` endpoints that verify what they are given, so placement was built
against a `ShardTransport` trait with an HTTP implementation on top of those —
and cross-node placement works today, on a real multi-node testnet, with no
libp2p changes at all.

The streaming libp2p protocol below is still worth doing (it removes an HTTP
hop between peers that are already connected, and gives repair a batched
`Have` probe). It is now a *second implementation of an existing trait* rather
than a prerequisite, which makes it independently schedulable.

### Fix the transport first

`request_response::cbor::Behaviour<FetchRequest, FetchResponse>` buffers an
Expand Down Expand Up @@ -93,11 +107,27 @@ Given a block needing `n` hosts, score each candidate peer:
```
score = reputation # c0mpute-verify::reputation, >= 0.9 required
* uptime_30d # >= 0.99 required (CIP-001)
* free_disk_factor # committed - used, normalised
* (1 / (1 + rtt_ms / 100)) # prefer near peers, weakly
* (0.9 + 0.1 / (1 + rtt_ms / 100)) # prefer near peers, weakly
```

Then select greedily under **diversity constraints**, in priority order:
Free disk is a hard filter rather than a score term — a peer either has room
for the shard or it does not.

**The latency weighting is deliberately narrower than this CIP first
specified.** A bare `1 / (1 + rtt/100)` factor makes a 400 ms peer score 20%
below a 1 ms one, which is enough for a fast flaky node to outrank a slow
reliable one. CIP-001 is explicit that availability drives durability and
latency does not, so the term is scaled into a band where it separates
otherwise-equal peers but cannot overturn a reputation gap. A unit test pins
this.

Greedy selection under a per-domain cap is **optimal, not heuristic**: "at
most `max_per_domain` from each domain" is a partition matroid, and greedy is
optimal over a matroid. So a `DiversityUnsatisfiable` result means no other
assignment would have worked either — no backtracking, and no better answer
being missed.

Select greedily under **diversity constraints**, in priority order:

1. No two shards of the same block on the same peer. (Hard.)
2. At most `floor(parity / 2)` shards per ASN — 2 of 14 for `standard`. (Hard.)
Expand Down
58 changes: 56 additions & 2 deletions docs/prds/005-repair-daemon.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
---
cip: 005
title: "Auto-repair daemon"
status: Draft
status: In progress
authors:
- anthony@profullstack.com
created: 2026-08-29
updated: 2026-08-29
implements: DIP-0012 (0012-storage-plugin.md) Phase 4
depends-on: 003, 004
blocks:
implementation:
implementation: PR #26 (c0mpute-placement::repair; `c0mpute storage repair`)
estimate: "3–4 weeks"
---

Expand Down Expand Up @@ -57,6 +57,40 @@ We chose the left column. The bill for that choice is paid here.

## Design

### What shipped, and what the implementation changed

The repair engine, election, flap tolerance and diversity-aware replacement
placement are implemented and driven by `c0mpute storage repair`. The
scheduled background daemon, gossip repair leases, signed attestations and the
bandwidth token bucket are **not** — see "Still outstanding" at the end.

Three things the design below did not anticipate, all found by running it:

1. **Election has to be bypassable.** `elect_repairer` picks among the block's
*holders*. An operator running `c0mpute storage repair` is usually not one,
so they could never win and every repair deferred forever. Election exists
to stop fourteen nodes doing the same job, not to stop anyone doing it, so
it is now a mode: honoured by the daemon, bypassed on explicit request.

2. **A dead peer still looks healthy in the catalog.** Reputation and uptime
are periodic measurements, not liveness. Repair happily selected the node
that had just died as the *destination* for the replacement — the repair
"succeeded" and the block stayed exactly as degraded. Replacement selection
now excludes every peer the block has ever pointed at, plus anything that
failed a probe this pass.

3. **That is not sufficient on its own.** A peer that died in an *earlier*
round is in the catalog, looks healthy, and is not probed at all because it
holds none of this block's shards. The first time we learn is when the
placement fails. So repair now selects spare candidates and fails over.
(Any subset of a valid selection is valid — the per-domain cap is a maximum
— so skipping a dead candidate cannot break diversity.)

The common thread is that **the catalog has no liveness signal**, and every
layer that assumes otherwise gets this wrong in a way that looks like success.
CIP-006's challenges are what eventually make peer health a measured fact
rather than a stale field.

### Who repairs?

Not the customer's client: a laptop that is closed for a week cannot be the
Expand Down Expand Up @@ -221,6 +255,26 @@ Defences:
election and leases, 1 week the repair path, 0.5 week attestations, 1 week
bandwidth control and storm defences, 0.5 week the chaos test harness.

## Still outstanding

Implemented: detection and classification, rendezvous election, flap-tolerant
condemnation, k-shard reconstruction with verification, minimal regeneration,
diversity-aware replacement with failover, attestation records, priority
ordering, and the storm cap on blocks per pass.

Not yet:

- **The scheduled daemon.** Repair runs on request today. The rolling
hourly scan needs somewhere to live — most naturally the worker supervisor.
- **Gossip repair leases.** Election alone prevents most duplicate work;
leases close the race when two nodes disagree about who is healthy.
- **Signed attestations.** The record exists and round-trips as JSON;
signing needs CoinPay DIDs, which arrive with CIP-006.
- **The bandwidth token bucket.** Repair is unthrottled, which is fine for
an operator-invoked pass and not for a background loop on a consumer uplink.
- **Batched `Have` probes.** One HEAD per shard, per CIP-003's HTTP transport.
Fine at this scale, too chatty for an hourly scan of millions of blocks.

## Open questions

- Should repair be *paid* after all, funded from the storage margin, to fix the
Expand Down
4 changes: 2 additions & 2 deletions docs/prds/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,9 @@ Delivering read/write network storage for c0mpute, implementing
|-----|-------|-----------|--------|
| [001](001-storage-program.md) | Storage program: durability model, tiers, and economics | — | In progress |
| [002](002-storage-http-api.md) | Storage HTTP API on the gateway | 001 | In progress |
| [003](003-shard-placement-transport.md) | Cross-node shard placement and streaming transport | 002 | Draft |
| [003](003-shard-placement-transport.md) | Cross-node shard placement and streaming transport | 002 | In progress |
| [004](004-metadata-durability.md) | Metadata durability: manifests, volumes, and the root pointer | 002 | Draft |
| [005](005-repair-daemon.md) | Auto-repair daemon | 003, 004 | Draft |
| [005](005-repair-daemon.md) | Auto-repair daemon | 003, 004 | In progress |
| [006](006-challenges-metering-payouts.md) | Storage challenges, metering, and provider payouts | 003, 004 | Draft |
| [007](007-c0mputefs-filesystem.md) | c0mputefs: mutable filesystem over immutable content | 004 | Draft |
| [008](008-write-path-consistency.md) | Write path: chunking, journal, and crash consistency | 007 | Draft |
Expand Down
3 changes: 3 additions & 0 deletions node/crates/c0mpute-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,15 @@ c0mpute-proto = { workspace = true }
c0mpute-update = { workspace = true }
c0mpute-secure-chat = { workspace = true }
c0mpute-store = { workspace = true }
c0mpute-placement = { workspace = true }
reqwest = { workspace = true }
c0mpute-gateway = { workspace = true }
axum = { workspace = true }
anyhow = { workspace = true }
tokio = { workspace = true }
tracing = { workspace = true }
clap = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
toml = { workspace = true }
rpassword = { workspace = true }
Expand Down
10 changes: 10 additions & 0 deletions node/crates/c0mpute-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
//! The plugin form mirrors the URL namespace: c0mpute.com/transcode,
//! c0mpute.com/coinpay, c0mpute.com/infernet.

mod peers;
mod storage;

use std::path::PathBuf;
Expand Down Expand Up @@ -356,6 +357,15 @@ fn maybe_self_update(cli: &Cli) {
fn maybe_self_update(_cli: &Cli) {}

fn main() -> Result<()> {
// Rust ignores SIGPIPE, so writing to a closed pipe returns EPIPE and the
// stdlib panics on it. For a CLI that prints lists that is a crash on
// `c0mpute storage ls | head`, which is ordinary shell usage. Restore the
// default disposition so the process exits quietly instead.
#[cfg(unix)]
unsafe {
libc::signal(libc::SIGPIPE, libc::SIG_DFL);
}

let cli = Cli::parse();

// Opportunistic self-update on any command (throttled), so c0mpute stays
Expand Down
Loading
Loading