CIP-005: auto-repair

docs/prds/005-repair-daemon.md

@@ -1,15 +1,15 @@
 ---
 cip: 005
 title: "Auto-repair daemon"
-status: Draft
+status: In progress
 authors:
   - anthony@profullstack.com
 created: 2026-08-29
 updated: 2026-08-29
 implements: DIP-0012 (0012-storage-plugin.md) Phase 4
 depends-on: 003, 004
 blocks:
-implementation:
+implementation: PR #26 (c0mpute-placement::repair; `c0mpute storage repair`)
 estimate: "3–4 weeks"
 ---
 
@@ -57,6 +57,40 @@ We chose the left column. The bill for that choice is paid here.
 
 ## Design
 
+### What shipped, and what the implementation changed
+
+The repair engine, election, flap tolerance and diversity-aware replacement
+placement are implemented and driven by `c0mpute storage repair`. The
+scheduled background daemon, gossip repair leases, signed attestations and the
+bandwidth token bucket are **not** — see "Still outstanding" at the end.
+
+Three things the design below did not anticipate, all found by running it:
+
+1. **Election has to be bypassable.** `elect_repairer` picks among the block's
+   *holders*. An operator running `c0mpute storage repair` is usually not one,
+   so they could never win and every repair deferred forever. Election exists
+   to stop fourteen nodes doing the same job, not to stop anyone doing it, so
+   it is now a mode: honoured by the daemon, bypassed on explicit request.
+
+2. **A dead peer still looks healthy in the catalog.** Reputation and uptime
+   are periodic measurements, not liveness. Repair happily selected the node
+   that had just died as the *destination* for the replacement — the repair
+   "succeeded" and the block stayed exactly as degraded. Replacement selection
+   now excludes every peer the block has ever pointed at, plus anything that
+   failed a probe this pass.
+
+3. **That is not sufficient on its own.** A peer that died in an *earlier*
+   round is in the catalog, looks healthy, and is not probed at all because it
+   holds none of this block's shards. The first time we learn is when the
+   placement fails. So repair now selects spare candidates and fails over.
+   (Any subset of a valid selection is valid — the per-domain cap is a maximum
+   — so skipping a dead candidate cannot break diversity.)
+
+The common thread is that **the catalog has no liveness signal**, and every
+layer that assumes otherwise gets this wrong in a way that looks like success.
+CIP-006's challenges are what eventually make peer health a measured fact
+rather than a stale field.
+
 ### Who repairs?
 
 Not the customer's client: a laptop that is closed for a week cannot be the
@@ -221,6 +255,26 @@ Defences:
 election and leases, 1 week the repair path, 0.5 week attestations, 1 week
 bandwidth control and storm defences, 0.5 week the chaos test harness.
 
+## Still outstanding
+
+Implemented: detection and classification, rendezvous election, flap-tolerant
+condemnation, k-shard reconstruction with verification, minimal regeneration,
+diversity-aware replacement with failover, attestation records, priority
+ordering, and the storm cap on blocks per pass.
+
+Not yet:
+
+- **The scheduled daemon.** Repair runs on request today. The rolling
+  hourly scan needs somewhere to live — most naturally the worker supervisor.
+- **Gossip repair leases.** Election alone prevents most duplicate work;
+  leases close the race when two nodes disagree about who is healthy.
+- **Signed attestations.** The record exists and round-trips as JSON;
+  signing needs CoinPay DIDs, which arrive with CIP-006.
+- **The bandwidth token bucket.** Repair is unthrottled, which is fine for
+  an operator-invoked pass and not for a background loop on a consumer uplink.
+- **Batched `Have` probes.** One HEAD per shard, per CIP-003's HTTP transport.
+  Fine at this scale, too chatty for an hourly scan of millions of blocks.
+
 ## Open questions
 
 - Should repair be *paid* after all, funded from the storage margin, to fix the

docs/prds/README.md

@@ -68,7 +68,7 @@ Delivering read/write network storage for c0mpute, implementing
 | [002](002-storage-http-api.md) | Storage HTTP API on the gateway | 001 | In progress |
 | [003](003-shard-placement-transport.md) | Cross-node shard placement and streaming transport | 002 | In progress |
 | [004](004-metadata-durability.md) | Metadata durability: manifests, volumes, and the root pointer | 002 | Draft |
-| [005](005-repair-daemon.md) | Auto-repair daemon | 003, 004 | Draft |
+| [005](005-repair-daemon.md) | Auto-repair daemon | 003, 004 | In progress |
 | [006](006-challenges-metering-payouts.md) | Storage challenges, metering, and provider payouts | 003, 004 | Draft |
 | [007](007-c0mputefs-filesystem.md) | c0mputefs: mutable filesystem over immutable content | 004 | Draft |
 | [008](008-write-path-consistency.md) | Write path: chunking, journal, and crash consistency | 007 | Draft |

node/crates/c0mpute-cli/src/storage.rs

@@ -72,6 +72,19 @@ pub enum StorageCmd {
     Info { hash: String },
     /// Re-read an object and verify every block against its hash.
     Verify { hash: String },
+    /// Rebuild shards lost to departed peers (CIP-005).
+    Repair {
+        /// Repair one object. Omit to sweep every object on this node.
+        hash: Option<String>,
+        /// Report what would be repaired without changing anything.
+        #[arg(long)]
+        dry_run: bool,
+        /// Treat an unreachable peer as gone immediately, skipping the grace
+        /// window. The window exists so a rebooting node is not repaired away;
+        /// override it only when you know a peer is really gone.
+        #[arg(long)]
+        now: bool,
+    },
     /// Delete an object and its shards.
     Rm {
         hash: String,
@@ -163,6 +176,9 @@ pub async fn run(cmd: StorageCmd, config_path: &std::path::Path) -> Result<()> {
         StorageCmd::Ls { quiet } => ls(config_path, quiet).await,
         StorageCmd::Info { hash } => info(config_path, &hash).await,
         StorageCmd::Verify { hash } => verify(config_path, &hash).await,
+        StorageCmd::Repair { hash, dry_run, now } => {
+            repair(config_path, hash.as_deref(), dry_run, now).await
+        }
         StorageCmd::Rm { hash, yes } => rm(config_path, &hash, yes).await,
         StorageCmd::Status => status(config_path).await,
         StorageCmd::Tiers => {
@@ -736,3 +752,126 @@ mod tests {
         }
     }
 }
+
+/// Rebuild shards lost to departed peers (CIP-005).
+///
+/// Explicit rather than elected: an operator running this has asked for the
+/// work directly, and is usually not one of the shard holders, so they could
+/// never win the rendezvous election that coordinates the background daemon.
+async fn repair(
+    config_path: &std::path::Path,
+    hash: Option<&str>,
+    dry_run: bool,
+    condemn_now: bool,
+) -> Result<()> {
+    let root = storage_root(config_path)?;
+    let catalog = peers::load(&root)?;
+    if catalog.is_empty() {
+        bail!(
+            "no storage peers configured — there is nowhere to repair from or to.\n\
+             Add peers with `c0mpute storage peer add`."
+        );
+    }
+    let storage = open(config_path).await?;
+    let local_id = format!("local:{}", root.display());
+    let repairer = c0mpute_placement::Repairer::new(
+        Arc::new(HttpTransport::default()),
+        Arc::new(RwLock::new(catalog)),
+        local_id,
+    )
+    .manual();
+
+    let objects = match hash {
+        Some(h) => vec![parse_hash(h)?],
+        None => storage.list().await?,
+    };
+    if objects.is_empty() {
+        println!("no objects on this node");
+        return Ok(());
+    }
+
+    let mut total_repaired = 0usize;
+    let mut total_shards = 0usize;
+    let mut total_lost = 0usize;
+
+    for object in objects {
+        let mut manifest = match storage.read_manifest(&object).await {
+            Ok(m) => m,
+            Err(e) => {
+                eprintln!("blake3:{}: unreadable manifest: {e:#}", object.to_hex());
+                continue;
+            }
+        };
+
+        if dry_run {
+            let plans = repairer.scan(&manifest, condemn_now).await?;
+            for plan in plans.iter().filter(|p| p.state.needs_repair()) {
+                println!(
+                    "blake3:{} block {} — {:?}, {} shard(s) missing {:?}",
+                    object.to_hex(),
+                    plan.block,
+                    plan.state,
+                    plan.missing.len(),
+                    plan.missing
+                );
+                total_repaired += 1;
+            }
+            continue;
+        }
+
+        let report = repairer
+            .repair_object(&mut manifest, 0, condemn_now)
+            .await?;
+        if report.blocks_repaired > 0 {
+            // The manifest now points at the new shard homes, so it has to be
+            // written back or the next read still looks for the dead peers.
+            storage.write_manifest(&manifest).await?;
+            println!(
+                "blake3:{} — repaired {} block(s), {} shard(s) regenerated",
+                object.to_hex(),
+                report.blocks_repaired,
+                report.shards_regenerated
+            );
+        }
+        for att in &report.attestations {
+            println!(
+                "  block {} shards {:?} → {}",
+                att.block,
+                att.shards_regenerated,
+                att.destinations.join(", ")
+            );
+        }
+        if report.blocks_lost > 0 {
+            eprintln!(
+                "blake3:{} — {} block(s) LOST: fewer than k shards remain, repair cannot help",
+                object.to_hex(),
+                report.blocks_lost
+            );
+        }
+        for f in &report.failures {
+            eprintln!("blake3:{}: {f}", object.to_hex());
+        }
+        total_repaired += report.blocks_repaired;
+        total_shards += report.shards_regenerated;
+        total_lost += report.blocks_lost;
+    }
+
+    if dry_run {
+        println!("\n{total_repaired} block(s) would be repaired (dry run)");
+        if total_repaired > 0 {
+            println!("re-run without --dry-run to rebuild them");
+        }
+    } else {
+        println!("\n{total_repaired} block(s) repaired, {total_shards} shard(s) regenerated");
+        if total_lost > 0 {
+            println!("{total_lost} block(s) unrecoverable");
+        }
+    }
+    if !condemn_now && total_repaired == 0 {
+        println!(
+            "note: peers unreachable for less than the grace window are left alone,\n\
+             so a node that is merely rebooting is not repaired away. Use --now to override."
+        );
+    }
+    Ok(())
+}

node/crates/c0mpute-placement/Cargo.toml

@@ -13,6 +13,7 @@ testing = []
 c0mpute-proto = { workspace = true }
 c0mpute-store = { workspace = true }
 anyhow = { workspace = true }
+blake3 = { workspace = true }
 async-trait = { workspace = true }
 futures = { workspace = true }
 reqwest = { workspace = true }

node/crates/c0mpute-placement/src/lib.rs

@@ -28,10 +28,17 @@
 
 pub mod distributed;
 pub mod peer;
+pub mod repair;
 pub mod select;
 pub mod transport;
 
 pub use distributed::{BlockHealth, BlockState, DistributedConfig, DistributedStorage};
 pub use peer::{FailureDomain, PeerCatalog, PeerInfo};
-pub use select::{Assignment, PlacementError, PlacementPolicy, score, select};
+pub use repair::{
+    FailureTracker, RepairAttestation, RepairConfig, RepairPlan, RepairReport, Repairer,
+    elect_repairer,
+};
+pub use select::{
+    Assignment, PlacementContext, PlacementError, PlacementPolicy, score, select, select_peers,
+};
 pub use transport::{HttpTransport, ShardTransport};