diff --git a/README.md b/README.md index dd50e57..c635cb4 100644 --- a/README.md +++ b/README.md @@ -136,8 +136,9 @@ lifecycle. See the unit header for GPU/role customization via `systemctl --user ``` . ├── docs/ -│ └── c0mpute-v1.md # v1 PRD (source of truth) -├── dips/ # design proposals +│ ├── c0mpute-v1.md # v1 PRD (source of truth) +│ └── prds/ # CIPs — per-phase PRDs (see prds/README.md) +├── dips/ # design proposals (the "why") ├── node/ │ └── crates/ # all Rust source — host + transcode workload │ ├── c0mpute-cli/ # produces `c0mpute` diff --git a/dips/0012-no-storage-network.md b/dips/0012-no-storage-network.md index 6521810..c15be10 100644 --- a/dips/0012-no-storage-network.md +++ b/dips/0012-no-storage-network.md @@ -1,17 +1,30 @@ --- dip: 0012 title: "c0mpute is compute-only; we don't run a storage network" -status: Accepted +status: Superseded authors: - anthony@profullstack.com created: 2026-05-03 -updated: 2026-05-03 +updated: 2026-08-29 discussion: implementation: supersedes: -superseded-by: +superseded-by: DIP-0012 (0012-storage-plugin.md) --- +> **Superseded.** This is draft v1 of DIP-0012 and no longer describes the +> project's position. It was withdrawn in favour of +> [`0012-storage-plugin.md`](0012-storage-plugin.md), whose motivation table +> records this draft and why it was dropped — the file simply never had its +> status updated, leaving two `Accepted` DIP-0012s on disk asserting opposite +> things. +> +> **c0mpute hosts files.** See the storage-plugin DIP for the current design, +> and [`docs/prds/`](../docs/prds/README.md) (CIP-001 onward) for the delivery +> plan. The cost analysis below is still worth reading: the five structural +> overheads it identifies are real, and CIP-001 and CIP-005 are largely +> answers to them. + ## Summary c0mpute is a **compute marketplace**, not a storage network. Customers diff --git a/dips/0012-storage-plugin.md b/dips/0012-storage-plugin.md index e9f808f..60166e2 100644 --- a/dips/0012-storage-plugin.md +++ b/dips/0012-storage-plugin.md @@ -258,6 +258,21 @@ egress as the differentiator. **No claim of "cheaper than R2."** ## Out of scope -- Filesystem-style mutable objects. Content-addressed, immutable. +- ~~Filesystem-style mutable objects. Content-addressed, immutable.~~ + **Amended 2026-08-29:** c0mpute ships a read/write POSIX mount. See + [CIP-007](../docs/prds/007-c0mputefs-filesystem.md). The immutability + property is preserved where it matters — blocks, manifests, and snapshot + nodes are still content-addressed and immutable. Mutability is confined to + advancing one signed 32-byte root pointer per volume + ([CIP-004](../docs/prds/004-metadata-durability.md)); the filesystem is a + naming layer above that, not a change to the storage layer. - Permanent / Arweave-style storage. - IPFS interop — could add later as a read-only adapter. + +## Delivery plan + +Phases 1–5 above are elaborated as CIPs in +[`docs/prds/`](../docs/prds/README.md). CIP-001 supersedes the pricing figures +in this DIP: tiered pricing at $0.0035/GB-month for RS 10/14 `standard`, based +on an expansion-factor advantage over Storj rather than on the $0.008 estimated +here. diff --git a/dips/README.md b/dips/README.md index 00a5dde..9daf47a 100644 --- a/dips/README.md +++ b/dips/README.md @@ -5,6 +5,13 @@ crosses a project-policy line, or is hard to undo once shipped. DIPs are where we record the *why* behind durable decisions so future contributors don't have to reverse-engineer them from git history. +> **DIP or CIP?** A DIP records *why we decided something*. A +> [CIP](../docs/prds/README.md) — c0mpute Improvement Protocol, in +> `docs/prds/` — is a PRD for a shippable phase: scope, API surface, +> acceptance criteria, estimate. A CIP implements a DIP and cites it. If a CIP +> finds its governing DIP is wrong, the fix is a new DIP, not a CIP that +> quietly contradicts one. Numbering is independent between the two. + If a change is small, local, and obvious from the diff — just open a PR. DIPs are for the things you'd want to read about a year from now. @@ -105,7 +112,8 @@ superseded-by: | 0009 | Mojo for GPU/kernel-shaped compute (when applicable) | Accepted | | 0010 | Operator-run seed nodes for libp2p Kad-DHT bootstrap | Accepted | | 0011 | No central backend; libp2p + CoinPay are source of truth | Accepted | -| 0012 | c0mpute is compute-only; storage is BYOS | Accepted | +| 0012 | c0mpute hosts files: Reed-Solomon 10/14, compute-locality value | Accepted | +| 0012 | ~~c0mpute is compute-only; storage is BYOS~~ (draft v1) | Superseded by `0012-storage-plugin.md` | | 0013 | Position: GPU batch-compute marketplace; 5–8× cheaper niche | Accepted | | 0014 | Public /status page + status-aggregator service | Accepted | | 0015 | Hosting vertical: censorship-resistant static sites | Draft | diff --git a/docs/prds/000-template.md b/docs/prds/000-template.md new file mode 100644 index 0000000..a2dca7d --- /dev/null +++ b/docs/prds/000-template.md @@ -0,0 +1,62 @@ +--- +cip: 000 +title: "Short imperative title — what ships" +status: Draft +authors: + - you@example.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-NNNN +depends-on: +blocks: +implementation: +estimate: +--- + +## Summary + +One paragraph. What is being built, and what can a user do at the end of it +that they couldn't before? A reviewer should be able to read this and decide +whether to keep reading. + +## Goals + +Bulleted, concrete, testable. "Fast" is not a goal; "p99 read latency under +400 ms for a 4 MiB block" is. + +## Non-goals + +What this phase deliberately leaves undone, especially things a reader will +otherwise assume are included. Point at the CIP that covers each one. + +## Design + +The actual proposal. Be specific enough that someone else could build it: + +- New API surface (routes, CLI flags, config keys, on-disk formats) +- Data structures and wire formats +- Failure modes and what happens in each +- Which crates change, and roughly how + +## Acceptance criteria + +A numbered checklist a reviewer can actually run. Each item is a command, a +test, or an observable behaviour — not a feeling. + +1. `cargo test -p c0mpute-store` passes with N new tests covering X. +2. … + +## Risks + +What could make this take twice as long, or ship broken. Include the mitigation +for each, or say plainly that there isn't one yet. + +## Estimate + +A range, with the assumption behind it (one engineer? two? familiar with the +codebase?). Break it down if the phase has distinct chunks. + +## Open questions + +Things intentionally unresolved. These should shrink to zero before status +moves to Approved. diff --git a/docs/prds/001-storage-program.md b/docs/prds/001-storage-program.md new file mode 100644 index 0000000..366e950 --- /dev/null +++ b/docs/prds/001-storage-program.md @@ -0,0 +1,280 @@ +--- +cip: 001 +title: "Storage program: durability model, tiers, and economics" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 (0012-storage-plugin.md) +depends-on: +blocks: 002, 003, 004, 005, 006, 007 +implementation: +estimate: "1 week (analysis + simulation, no production code)" +--- + +## Summary + +This CIP sets the parameters every other storage CIP builds on: which +redundancy scheme backs which tier, what we pay providers, what we charge +customers, and what durability we can honestly claim. It produces no shipping +code — it produces numbers that the rest of the program is not allowed to +contradict. + +The headline decision: **Reed-Solomon 10/14 is the default, replication is a +tier not a default, and the retail price is $0.0035/GB-month** — which +undercuts Storj, the only p2p network that is actually comparable on +retrieval latency. + +## Resolve the DIP-0012 collision first + +The repo currently contains **two files numbered DIP-0012, both marked +`Accepted`, both dated 2026-05-03, asserting opposite things**: + +- `dips/0012-no-storage-network.md` — "c0mpute is compute-only; we don't run a + storage network." +- `dips/0012-storage-plugin.md` — "c0mpute hosts files: Reed-Solomon 10/14." + +Neither declares `supersedes` or `superseded-by`. This is a live hazard: the +next contributor to read `dips/` in lexical order finds the wrong one first. + +**Action, in the PR that lands this CIP:** mark `0012-no-storage-network.md` +as `Superseded`, point its `superseded-by` at the storage-plugin DIP, and fix +the DIP index table which currently reads "c0mpute is compute-only; storage is +BYOS" for 0012. The storage-plugin DIP's own motivation table already records +that this position was drafted and withdrawn — the file just never got its +status updated. + +## Goals + +- Pick `(k, n)` per tier, with the durability arithmetic written down. +- Set provider payout rates and customer retail prices that leave a real + margin, including repair traffic. +- State a defensible "cheapest" claim that survives someone checking it. +- Define what a node must promise before it is allowed to hold shards. + +## Non-goals + +- Cold/archival tier. We are structurally bad at it (see + `docs/storage-pricing-scenarios.md` scenario 4) and Glacier Deep Archive at + $0.00099/GB is not a fight worth having. Revisit only if a customer pays. +- Token, staking, or collateral design. Payouts ride CoinPay (DIP-0007); + slashing is reputation-based, not bonded. + +## Design + +### Why erasure coding, not 3-copy replication + +The instinct to "keep 3 copies" is the single most expensive decision +available here, because the redundancy factor *is* the cost of goods: + +| Scheme | Raw GB per usable GB | Tolerates | Repair amplification | +|---|---|---|---| +| 3-copy replication | **3.00x** | 2 losses | **1x** (copy a survivor) | +| RS 10/14 (shipped) | **1.40x** | 4 losses | **10x** (fetch k shards) | +| RS 20/32 | 1.60x | 12 losses | 20x | +| Storj RS 29/80 | 2.76x | 51 losses | 29x | + +Replication and erasure coding trade the same two costs in opposite +directions: **replication is cheap to repair and expensive to store; erasure +coding is cheap to store and expensive to repair.** Everything below follows +from that sentence. + +Against Storj specifically, our expansion factor is the structural advantage. +At an identical provider payout rate, RS 10/14 costs **1.97x less** per usable +GB than RS 29/80. That is not a cleverness advantage that Storj can copy back +— widening their code would cut their own durability margin, which is what +their slow-repair architecture spends it on. + +### Durability is bounded by per-node availability, not by parity + +This is the part that is easy to get wrong. RS 10/14 needs 10 of 14 shards. If +each shard host is independently available with probability `p`, an object is +readable with probability `P(X >= 10)` where `X ~ Binomial(14, p)`: + +Computed by `scripts/storage-durability-sim.py`: + +| Per-node availability | RS 10/14 | 3-copy | RS 20/32 | Storj RS 29/80 | +|---|---|---|---|---| +| 0.90 | 9.2e-3 (2.0 nines) | 1.0e-3 (3.0) | 5.5e-6 (5.3) | 1.6e-32 (31.8) | +| 0.95 | 4.3e-4 (3.4 nines) | 1.3e-4 (3.9) | 1.7e-9 (8.8) | 1.6e-47 (46.8) | +| 0.99 | 1.9e-7 (6.7 nines) | 1.0e-6 (6.0) | 2.9e-18 (17.5) | 2.2e-83 (82.7) | +| 0.999 | 2.0e-12 (11.7 nines) | 1.0e-9 (9.0) | 3.4e-31 (30.5) | 2.8e-135 (134.5) | + +**RS 10/14 on 95%-available consumer nodes yields roughly three nines, not +eleven.** The "11 nines" figure in `docs/storage-pricing.csv` is only reachable +with ~99.9% per-node availability *and* fast repair. + +Two further readings of that table are worth internalising, because both cut +against intuition: + +- **At low availability, 3-copy beats RS 10/14** (3.9 nines vs 3.4 at p=0.95). + Needing 1-of-3 is a weaker demand than 10-of-14. Wide erasure codes only pull + ahead once nodes are individually reliable. This is another argument for the + `hot` tier being replicated rather than coded. +- **Storj's 29/80 is not waste, it is a different bet.** 46 nines at p=0.95 + means they can repair *lazily* — batch it, run it cheaply, tolerate a node + being gone for weeks. Our 1.4x expansion buys the cost advantage by spending + their safety margin, which means **we are obligated to repair fast.** CIP-005 + is therefore not a nice-to-have that follows the launch; it is the load-bearing + component of this entire cost model. If repair is slow or broken, RS 10/14 is + the wrong code and we will lose data that Storj would not have lost. + +Two consequences for implementation: + +1. **Placement must be reputation-gated.** Shards go only to nodes with + `reputation >= 0.9` (`c0mpute-verify::reputation`) and 30-day uptime + `>= 99%`. A node below that line can still run compute; it just doesn't get + paid to hold data. CIP-003 enforces this. +2. **`(k, n)` must be configurable, not hard-coded.** `c0mpute-store::erasure` + already takes `k` and `parity` as arguments and only the *defaults* are + 10/14 — good. `ObjectManifest` already persists `k` and `parity` per object, + so a future parameter change is not a migration. Keep it that way. + +Correlated failure is not modelled above and is the thing most likely to bite: +14 shards behind one ISP, one power grid, or one hosting provider are not 14 +independent samples. CIP-003 requires ASN and region diversity in placement for +this reason, and the availability figures should be read as an upper bound +until CIP-005's repair loop is measured in production. + +### Tiers + +Three tiers, matching DIP-0012 v3, with the redundancy scheme now pinned: + +| Tier | Scheme | Expansion | Best for | Retail $/GB-mo | +|---|---|---|---|---| +| `hot` | 3-copy replication | 3.0x | Small files, metadata, DB-adjacent, latency-sensitive reads | **$0.006** | +| `standard` | RS 10/14 | 1.4x | Default. Bulk data, media, datasets | **$0.0035** | +| `critical` | RS 20/32 | 1.6x | Irreplaceable data, long retention | **$0.005** | + +`hot` exists precisely because replication's 1x repair amplification and +single-peer reads make it the right tool for small, frequently-read, frequently- +rewritten objects — which is exactly what a filesystem's metadata and a +database's pages look like. CIP-007 places filesystem metadata on `hot` for +this reason. So the three-copy instinct was right, just for a narrower job than +"everything". + +Encryption is orthogonal to tier and is covered by CIP-011; every tier can be +client-encrypted. + +### Provider economics + +Providers are paid per raw GB actually held, per month, prorated hourly, plus +per GB served to paying customers. + +| Line item | Rate | Notes | +|---|---|---| +| Storage held | **$0.0015 / raw GB / month** | Matches Storj's node rate, so no reason to prefer them | +| Customer egress served | **$0.002 / GB** | Paid only for bytes a customer actually pulled | +| Repair egress | **$0 — unpaid obligation** | See below. This is load-bearing. | +| Ingress | $0 | Writes are free to accept | + +**Repair egress must be unpaid, or the pricing collapses.** Work the numbers: +at monthly node churn `c`, the bytes that must be *read* to rebuild what was +lost, per usable GB, is `k × expansion × c` = `10 × 1.4 × c`. At 5% monthly +churn that is 0.7 GB of transfer per usable GB per month. Paying the egress +rate on it would add $0.0014/GB-month to a cost base of $0.0021 — a 67% COGS +increase that erases the margin outright. + +Storj resolves this the same way: nodes are paid for customer egress, not for +repair traffic. So this is standard, not sharp practice — but it must be +**explicit in the provider terms**, because it means a node with a metered or +tightly capped residential uplink is a bad fit and will feel cheated later. +Provider onboarding (CIP-006) states an uplink expectation up front. + +### Customer economics and the "cheapest" claim + +COGS per usable GB-month, at the $0.0015 provider rate: + +| Tier | Expansion | COGS | Retail | Gross margin | +|---|---|---|---|---| +| `standard` (RS 10/14) | 1.4x | $0.0021 | $0.0035 | **40%** | +| `hot` (3-copy) | 3.0x | $0.0045 | $0.006 | 25% | +| `critical` (RS 20/32) | 1.6x | $0.0024 | $0.005 | 52% | + +Egress: **$0 internal** (to c0mpute jobs — the compute-locality argument that +DIP-0012 rests on), **$0.004/GB to the public internet**, undercutting Storj's +$0.007. + +Now the honest framing of "cheapest of any p2p storage", because the claim as +stated does not survive contact with the price sheet: + +| Network | $/GB-mo | Why it is or isn't comparable | +|---|---|---| +| Filecoin | $0.0001 | Archival. Deal minimums, 180-day terms, retrieval in minutes-to-hours | +| Lighthouse | $0.0003 | One-time payment, Filecoin-backed, no mutable access | +| Sia | $0.001 | 3-month renewal commitment; tiny network; volatile | +| **c0mpute `standard`** | **$0.0035** | Hot, mountable, read/write, no commitment | +| Storj DCS | $0.004 | Hot, S3, no commitment — **the real comparable** | +| Crust | $0.001 | Small network, native API only | + +**We cannot be cheaper than Filecoin or Sia, and should never claim to be.** +They are cold or commitment-bound; a POSIX read/write mount over them is not a +product anyone can use. The claim that is both true and marketable: + +> The cheapest p2p storage you can actually mount — no commitment, no +> minimums, no token, and $0 egress to compute. + +Against Storj — the only network offering comparable retrieval latency and no +commitment — we are **12.5% cheaper on storage and 43% cheaper on egress**, and +the margin to do it comes from the expansion-factor advantage, not from +underpaying providers. + +### What a storage node must promise + +Enforced by CIP-003 placement and CIP-006 challenges: + +- 30-day uptime >= 99%, reputation >= 0.9 +- Unmetered or high-cap uplink; repair traffic is unpaid +- Minimum 100 GB committed, minimum 30-day intent +- Responds to byte-range challenges within 5s, >= 99% pass rate +- Graceful exit: announce, let repair drain your shards, then leave. Nodes that + vanish without announcing take a reputation hit that gates them out of + placement. + +## Acceptance criteria + +1. `dips/0012-no-storage-network.md` has `status: Superseded` and + `superseded-by: DIP-0012 (0012-storage-plugin.md)`; the DIP index row for + 0012 reads the storage-plugin title. +2. A committed simulation (`scripts/storage-durability-sim.*`) reproduces the + availability table above and is re-runnable with different `(k, n, p)`. +3. `docs/storage-pricing.csv` rows for c0mpute are updated to the tier prices + here, and gain a `tier` column. +4. `docs/storage-pricing-scenarios.md` is re-run against the new prices. +5. No other CIP in this program cites a price or `(k, n)` not listed here. + +## Risks + +- **Consumer-node availability comes in under 99%.** Then `standard` delivers + ~3 nines and the durability claim has to be restated or `(k, n)` widened. + *Mitigation:* the sim is parameterised; gate placement on measured uptime + from day one rather than assuming it. +- **Churn is much higher than 5%/month.** Repair traffic grows linearly and + providers with capped uplinks quit, which raises churn further. This is the + one genuinely reflexive failure mode in the design. *Mitigation:* measure + churn before public launch; if it exceeds 10%/month, move `standard` to + RS 20/32 and reprice. +- **Storj cuts prices in response.** Their expansion factor means they'd be + cutting into node pay to do it. *Mitigation:* none needed; the structural + advantage is real, but don't build a plan that requires them to stand still. +- **Nobody supplies disk at $0.0015/GB.** *Mitigation:* the rate matches an + existing market clearing price; if supply is short, raise payout and margin + absorbs it down to ~$0.0021 before `standard` goes underwater. + +## Estimate + +**1 week.** Analysis, the durability simulation, and the pricing doc updates. +No production code. Do not start CIP-002 before this is Approved — every later +phase hard-codes numbers from this document. + +## Open questions + +- Does `hot` (3-copy) need its own placement policy, or does CIP-003's + reputation gate suffice at n=3? Losing 2 of 3 is far likelier than losing 5 + of 14. +- Should `critical` be RS 20/32 or RS 16/24? 20/32 doubles read fan-out versus + `standard`, which may hurt more than the durability helps. +- Minimum billable object size. At 4 KiB, 14 shards of ~300 bytes each plus a + manifest is mostly overhead — the classic small-file problem. CIP-007 packs + small files into larger blocks, but the *billing* floor still needs a number. diff --git a/docs/prds/002-storage-http-api.md b/docs/prds/002-storage-http-api.md new file mode 100644 index 0000000..efb3b05 --- /dev/null +++ b/docs/prds/002-storage-http-api.md @@ -0,0 +1,227 @@ +--- +cip: 002 +title: "Storage HTTP API on the gateway" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 (0012-storage-plugin.md) Phase 2 +depends-on: 001 +blocks: 003, 004, 012 +implementation: +estimate: "1.5–2 weeks" +--- + +## Summary + +Expose the already-working `c0mpute-store::Storage` engine over HTTP so +something other than a unit test can put and get an object. Single node only — +all 14 shards land on the local disk and `host_hint` stays `None`. This is the +smallest change that turns shipped library code into a usable service. + +## Motivation + +`c0mpute-store` has erasure coding, manifests, integrity verification, and nine +passing tests. The gateway (`c0mpute-gateway/src/lib.rs`) currently exposes +exactly two routes: + +```rust +Router::new() + .route("/healthz", get(healthz)) + .route("/chunks/{hash}", get(chunk_handler)) +``` + +There is no way to reach the storage engine from outside the process. Every +later phase — placement, repair, the filesystem, S3 — needs this surface, and +none of them can be built or tested without it. + +## Goals + +- PUT/GET/DELETE/HEAD an object by content hash over HTTP. +- PUT/GET a single shard, which is the primitive CIP-003 uses for placement and + CIP-005 uses for repair. +- Streaming request and response bodies — no `Vec` of a whole object. +- Signed-request auth on writes, per DIP-0007. +- Per-tier `(k, n)` selection from CIP-001. + +## Non-goals + +- Cross-node placement (CIP-003). `host_hint` is `None` throughout. +- Repair (CIP-005), challenges or billing (CIP-006). +- Mutable paths, directories, or names (CIP-007). Objects are content-addressed + and immutable here. +- S3 wire compatibility (CIP-012). + +## Design + +### Routes + +Added to `c0mpute-gateway`, behind the `storage` role being enabled: + +``` +PUT /storage/v1/objects/{object_hash} store an object +GET /storage/v1/objects/{object_hash} reconstruct and stream it back +HEAD /storage/v1/objects/{object_hash} existence + length, no body +DELETE /storage/v1/objects/{object_hash} drop manifest + unreferenced shards + +PUT /storage/v1/shards/{shard_hash} accept one shard (peer placement) +GET /storage/v1/shards/{shard_hash} serve one shard +HEAD /storage/v1/shards/{shard_hash} do you hold it? + +GET /storage/v1/manifests/{object_hash} the manifest as JSON +``` + +`{object_hash}` is `blake3:`; a bare hex string is also accepted. + +### Object PUT + +```http +PUT /storage/v1/objects/blake3:9f86d0... HTTP/1.1 +X-Coinpay-Auth: base64url(envelope) +X-C0mpute-Tier: standard ; hot | standard | critical, default standard +Content-Type: application/octet-stream +Content-Length: 1048576 +``` + +The client commits to the hash in the URL. The server streams the body to a +temp file, hashes as it goes, and **rejects with 422 if the computed hash does +not match the URL**. This is what makes the store trustworthy without trusting +the uploader — it is the same property `ChunkStore::get` already enforces on +read. + +Response `201 Created` returns the manifest as JSON (the `ObjectManifest` +struct already derives `Serialize`). + +Idempotent: PUT of an object that already exists returns `200 OK` with the +existing manifest and writes nothing. + +### Streaming is mandatory, not an optimisation + +`Storage::put(&self, data: &[u8])` takes a full slice, and `get` returns +`Vec`. For a filesystem backend that is untenable — a 4 GiB file would mean +a 4 GiB allocation on both ends, and the c0mpute worker rigs this is aimed at +also run inference. + +This CIP adds streaming variants alongside the existing ones: + +```rust +impl Storage { + /// Consume an AsyncRead, hashing and RS-encoding in fixed blocks. + pub async fn put_stream( + &self, reader: R, expected: Option, tier: Tier, + ) -> Result; + + /// Produce an AsyncRead that reconstructs lazily, block by block. + pub fn get_stream(&self, object_hash: &Hash) + -> Result; + + /// Byte-range read. Needed by CIP-007 for random-access files. + pub async fn get_range( + &self, object_hash: &Hash, offset: u64, len: u64, + ) -> Result>; +} +``` + +Both are built on a **block layer**: an object is split into fixed-size blocks +(default 4 MiB, recorded in the manifest) and each block is independently +RS-encoded into `n` shards. Consequences, all of which later CIPs depend on: + +- Memory is bounded by block size, not object size. +- `get_range` fetches only the blocks a range touches, so random access does + not read the whole file. CIP-007 cannot exist without this. +- A single damaged block is repairable without touching the rest of the object. + +This means `ObjectManifest` grows a block dimension. Bump it to a versioned +format now, while nothing depends on it: + +```rust +pub struct ObjectManifest { + pub version: u8, // NEW: 2 + pub object_hash: Hash, + pub original_len: u64, + pub block_size: u32, // NEW: bytes per block, default 4 MiB + pub k: u8, + pub parity: u8, + pub tier: Tier, // NEW + pub blocks: Vec, // NEW: replaces flat `shards` +} + +pub struct BlockEntry { + pub index: u32, + pub len: u32, // pre-padding plaintext length + pub shards: Vec, // the existing struct, unchanged +} +``` + +Version 1 manifests (flat `shards`, single implicit block) still parse — a +`#[serde(default)]` shim maps them to a one-block v2. There is no production +data to migrate, but the shim keeps the existing tests meaningful. + +### Auth + +Writes (`PUT`, `DELETE`) require the DIP-0007 signed-request envelope in +`X-Coinpay-Auth`. Reads of `standard`/`hot` objects are unauthenticated — +knowing a blake3 hash is itself the capability. `private`, client-encrypted +objects (CIP-011) are also served unauthenticated because the bytes are +ciphertext; confidentiality comes from the key, not the ACL. + +Shard endpoints (`PUT /shards/...`) require auth from a peer whose DID is a +known network member, so a stranger cannot fill our disk. Rate-limited per DID. + +### Errors + +| Code | When | +|---|---| +| 400 | Malformed hash, bad tier, missing length | +| 401 | Missing or invalid envelope on a write | +| 404 | No manifest, or shard not held | +| 409 | PUT in flight for the same hash | +| 413 | Object above `max_object_bytes` | +| 422 | Body hash != URL hash | +| 507 | Disk budget exhausted | + +## Acceptance criteria + +1. `curl -X PUT --data-binary @file` then `curl -O` round-trips a 1 GiB file + with a matching sha256, and the node's RSS stays under 200 MB throughout. +2. PUT with a deliberately wrong hash in the URL returns 422 and leaves nothing + on disk. +3. `GET` with `Range: bytes=1000000-1004095` returns exactly 4096 bytes and + fetches only the blocks covering that range (assert via a shard-read counter + in tracing). +4. Deleting 4 of 14 shards of one block still serves the whole object; deleting + 5 returns 500 with a decode-shortage error naming the block. +5. Existing `c0mpute-store` tests pass unmodified; a v1 manifest fixture still + deserialises. +6. Writes without a valid envelope get 401; reads don't need one. +7. `c0mpute doctor` reports the storage role's disk budget and current usage. + +## Risks + +- **The manifest format change ripples.** Doing it now, before CIP-003/004/007 + exist, is deliberately the cheapest moment. *Mitigation:* version field plus + the v1 shim; land this before anything else consumes manifests. +- **`Storage` gains a second, near-duplicate code path.** *Mitigation:* make the + non-streaming `put`/`get` thin wrappers over the streaming ones rather than + maintaining both. +- **Blocks change dedup granularity.** Content-addressed shards dedup within a + block boundary only; a file that shifts by one byte re-encodes entirely. + Content-defined chunking would fix it and is deferred to CIP-007, which has + the write patterns to justify it. + +## Estimate + +**1.5–2 weeks.** Roughly: 3 days for the block layer and manifest v2, 3 days +for streaming put/get plus range reads, 2 days for routes and auth wiring, +2 days for tests and `doctor` integration. + +## Open questions + +- `max_object_bytes` default. 4 MiB blocks make large objects tractable, but + the manifest itself grows ~14 shard entries per block — a 1 TiB object means + ~262k blocks and a manifest in the tens of MB. CIP-004 needs to handle large + manifests, or block size must scale with object size. +- Does `hot` (3-copy) even use the RS path with k=1, or a separate replication + path? k=1/parity=2 is degenerate but correct RS, and reusing the code is + tempting. Measure before deciding. diff --git a/docs/prds/003-shard-placement-transport.md b/docs/prds/003-shard-placement-transport.md new file mode 100644 index 0000000..8a5f3eb --- /dev/null +++ b/docs/prds/003-shard-placement-transport.md @@ -0,0 +1,219 @@ +--- +cip: 003 +title: "Cross-node shard placement and streaming transport" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 (0012-storage-plugin.md) Phase 3 +depends-on: 002 +blocks: 005, 006 +implementation: +estimate: "3–4 weeks" +--- + +## Summary + +Spread each block's `n` shards across `n` distinct peers chosen for reputation +and failure-domain diversity, and make the libp2p transport capable of moving +them without loading whole shards into memory. This is the phase where +"distributed" stops being aspirational. + +## Motivation + +After CIP-002 the storage service works and stores every shard on one disk, +which provides no durability at all — one disk failure loses everything, and +the RS coding is pure overhead. `ShardEntry::host_hint` exists in the manifest +and is always `None`. + +The network layer is further along than it looks. `c0mpute-net` already has an +846-line libp2p swarm with Kademlia, gossipsub, mDNS, and a request-response +protocol at `/c0mpute/chunk-fetch/1.0.0` that is already keyed by hash: + +```rust +struct FetchRequest { chunk_hash: Hash } +enum FetchResponse { Ok { bytes: Vec }, NotFound } +``` + +That is exactly the right shape and exactly the wrong encoding, which is the +first thing this CIP fixes. + +## Goals + +- Select `n` peers per block, diverse by ASN and region, gated on reputation. +- Push shards to peers and fetch them back, streaming, with bounded memory. +- Populate `host_hint` and make reads fetch from peers. +- Degrade sanely: a read succeeds while up to `parity` hosts are unreachable. +- Publish and discover "who holds shard X" without a central index. + +## Non-goals + +- Repairing what's lost (CIP-005) — this phase detects and tolerates loss, + it does not fix it. +- Paying anyone (CIP-006). +- Geographic *pinning* or data-residency guarantees. Diversity is for + durability here, not for compliance. + +## Design + +### Fix the transport first + +`request_response::cbor::Behaviour` buffers an +entire response in a `Vec` before delivering it. With 4 MiB blocks and +RS 10/14 a shard is ~400 KiB, which is survivable; but the `hot` tier stores +whole 4 MiB blocks per replica, and `critical` reads fan out to 20 peers at +once. A node serving 50 concurrent reads would hold hundreds of MB in CBOR +buffers on a rig that is also running inference. + +Replace it with a **streamed protocol** at `/c0mpute/shard/1.0.0`, using +`libp2p-request-response` with a custom `Codec` that reads and writes framed +chunks straight to and from disk: + +```rust +enum ShardRequest { + Get { shard_hash: Hash }, + Put { shard_hash: Hash, object_hash: Hash, block: u32, index: u8, len: u32 }, + Have { shard_hashes: Vec }, // batched existence probe +} +``` + +Bodies stream in 64 KiB frames. The receiver hashes as it goes and rejects a +`Put` whose bytes don't match the declared `shard_hash` — same +commit-then-verify property as CIP-002's object PUT. + +Keep `/c0mpute/chunk-fetch/1.0.0` registered and working for one release so +older nodes interoperate, then drop it. Protocol IDs are a public surface +(DIP-0003 territory), so the version bump is deliberate. + +### Peer selection + +Given a block needing `n` hosts, score each candidate peer: + +``` +score = reputation # c0mpute-verify::reputation, >= 0.9 required + * uptime_30d # >= 0.99 required (CIP-001) + * free_disk_factor # committed - used, normalised + * (1 / (1 + rtt_ms / 100)) # prefer near peers, weakly +``` + +Then select greedily under **diversity constraints**, in priority order: + +1. No two shards of the same block on the same peer. (Hard.) +2. At most `floor(parity / 2)` shards per ASN — 2 of 14 for `standard`. (Hard.) +3. At most `floor(parity / 2)` shards per region. (Hard.) +4. Prefer peers not already holding shards of the same *object*. (Soft.) + +Constraint 2 is the one that matters and the one that will fail first. The +independence assumption behind CIP-001's durability table is worth nothing if +ten shards sit behind one residential ISP in one metro. If the network cannot +satisfy the constraints, **placement fails loudly rather than silently +degrading**: return 507 with which constraint could not be met. A small network +genuinely cannot store data durably, and pretending otherwise is how people +lose files. + +Bootstrap reality: with fewer than ~30 storage nodes across ~5 ASNs, +`standard` placement will legitimately fail. Until then the network runs in +`hot` (n=3) which needs only 3 diverse peers, and the CLI says so plainly. + +### Discovery: who holds this shard? + +Two mechanisms, belt and braces: + +- **The manifest is the primary index.** `host_hint` names the peer per shard. + Reads go straight there. This is fast and needs no lookup. +- **Kad DHT is the fallback.** Providers announce `provide(shard_hash)` on the + existing Kademlia behaviour. When a `host_hint` is stale — the peer moved, + or repair relocated the shard — the reader falls back to + `kad_find_node`/`get_providers`. `Swarm::kad_find_node` already exists. + +Hints go stale constantly and that is fine; they are hints. The DHT is the +source of truth, the manifest is the cache. + +### Read path + +To read block `i`: + +1. Take the `n` `host_hint`s from the manifest. +2. Fire `Get` to all `n` concurrently. Accept the first `k` that return. +3. Cancel the stragglers. RS decode. Verify against the block hash. +4. If fewer than `k` return within the deadline, resolve missing shards via the + DHT and retry once. +5. If still short, return an error naming the block and how many shards were + found — and enqueue a repair (CIP-005). + +Requesting all `n` and taking the first `k` costs `n/k` = 1.4x read bandwidth +in exchange for cutting tail latency to the `k`-th fastest peer instead of the +slowest of a chosen `k`. On a network of consumer nodes with 200–500 ms +latencies that trade is clearly worth it. `critical` at 32 shards makes it +worse (1.6x), which is another argument for the 16/24 option in CIP-001's open +questions. + +### Write path + +1. Encode the block into `n` shards locally (already implemented). +2. Select `n` peers. +3. `Put` all `n` concurrently, with a deadline. +4. **Acknowledge the write once `k + ceil(parity/2)` shards are confirmed** — + 12 of 14 for `standard`. Full `n` placement continues in the background. + This bounds write latency by the 12th-fastest peer rather than the slowest, + while still leaving the object readable if the two stragglers never land. +5. Record `host_hint` for every confirmed shard; hand unconfirmed ones to the + repair queue. + +The write is durable at step 4 in the sense that the data survives `parity/2` +further failures. That is the honest definition and it is what `fsync` will map +onto in CIP-008. + +## Acceptance criteria + +1. A 5-node local testnet (mDNS, already supported) stores an object and each + node holds strictly fewer than `k` shards of any block — verified by + inspecting each node's chunk store. +2. Killing any 4 of 14 shard hosts still serves the object; killing 5 fails + with an error naming the block. +3. Placement on a 3-node network with `standard` returns 507 naming the + unsatisfiable diversity constraint, and does not write a partial object. +4. Streaming a 1 GiB object across the testnet keeps every node's RSS under + 300 MB. +5. A stale `host_hint` (peer restarted with a new address) still resolves via + the DHT, with a tracing event recording the fallback. +6. Write acknowledges after 12 of 14 shards on a network where 2 peers are + artificially delayed by 10s, and the remaining 2 land afterwards. +7. Old nodes speaking `/c0mpute/chunk-fetch/1.0.0` can still fetch chunks. + +## Risks + +- **Not enough diverse peers at launch.** The most likely blocker, and it's a + supply problem, not a code problem. *Mitigation:* `hot` tier at n=3 works on + a tiny network; operator-run seed nodes (DIP-0010) provide initial ASN + diversity; the CLI reports how far the network is from supporting `standard`. +- **ASN lookup needs a data source.** A bundled IP-to-ASN table goes stale; + a lookup service is a central dependency, which DIP-0011 forbids. + *Mitigation:* ship an embedded table refreshed per release, degrade to /16 + prefix diversity when unknown. Prefix diversity is weaker but never wrong. +- **NAT.** Consumer nodes are behind NAT and libp2p hole-punching is not + configured in the current swarm. Without it, a large fraction of "diverse" + peers are simply unreachable. *Mitigation:* enable DCUtR + relay in the + swarm as part of this CIP; treat relay-only peers as lower-scored, since + relayed bandwidth is somebody else's cost. +- **Sybil placement.** One operator running 14 nodes across 14 VPS providers + defeats diversity while satisfying every constraint. *Mitigation:* out of + scope here; CIP-006's challenge economics and reputation are the lever, and + this is an accepted residual risk for v1. + +## Estimate + +**3–4 weeks.** ~1 week for the streaming protocol and codec, 1 week for +selection and diversity constraints, 0.5 week for DHT provider records, 1 week +for the read/write paths with partial-failure handling, 0.5 week for the +testnet harness. + +## Open questions + +- Should `Have` batching be a separate protocol or folded into Kad provider + records? Batched probes are much cheaper for CIP-005's repair scans. +- Relay-assisted peers: count them toward diversity or not? They are reachable + but their bandwidth is a third party's. +- Does write-ack at `k + parity/2` need to be tier-configurable? `critical` + users may want full `n` before ack. diff --git a/docs/prds/004-metadata-durability.md b/docs/prds/004-metadata-durability.md new file mode 100644 index 0000000..7f4d3bc --- /dev/null +++ b/docs/prds/004-metadata-durability.md @@ -0,0 +1,229 @@ +--- +cip: 004 +title: "Metadata durability: manifests, volumes, and the root pointer" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 (0012-storage-plugin.md) — resolves its "manifest hosting durability" open question +depends-on: 002 +blocks: 005, 006, 007, 012 +implementation: +estimate: "3–4 weeks" +--- + +## Summary + +Give manifests somewhere durable to live, and give a customer a single stable +name for their data that survives every write. This solves the problem DIP-0012 +flagged and left open: + +> A manifest = a small JSON saying "these 14 shards on these 14 hosts make +> object X." If the manifest is lost, the data is unrecoverable even though +> shards exist. + +The answer is a **volume**: a named, mutable root pointer, signed by a CoinPay +DID, whose value is the hash of an immutable metadata snapshot. Everything +mutable in the entire system reduces to advancing that one pointer. + +## Motivation + +Right now a manifest is a JSON file at `manifests/.json` on whichever +node happened to run the PUT. That node is a single point of total data loss, +and there is no way to enumerate "my objects" at all. + +This is also the pivot the read/write filesystem depends on. Content is +immutable and content-addressed; a filesystem is mutable and path-addressed. +The only way to build the second from the first is a mutable pointer that names +an immutable tree. Get this layer right and CIP-007 is mostly bookkeeping. Get +it wrong and no amount of FUSE work will save it. + +## Goals + +- No single node's loss can orphan reachable data. +- A customer has a stable identifier for a mutable dataset. +- Root updates are atomic, ordered, and attributable to a DID. +- Recovery from a total client loss needs only the DID key. +- Bounded metadata cost for large datasets. + +## Non-goals + +- Concurrent multi-writer resolution (CIP-010 — this CIP assumes one writer + at a time and detects, but does not merge, conflicts). +- POSIX semantics: inodes, permissions, directory entries (CIP-007). +- Encryption of metadata (CIP-011). + +## Design + +### Three layers + +``` + Root pointer volume id -> snapshot hash mutable, signed, tiny + │ (CoinPay-anchored) + ▼ + Snapshot an immutable metadata tree content-addressed + │ (object index / later: inodes) stored as a `hot` object + ▼ + Manifests object hash -> blocks -> shards content-addressed + │ stored as `hot` objects + ▼ + Shards the actual bytes RS-coded across n peers +``` + +Every layer below the root is immutable and content-addressed, so it inherits +the durability of CIP-003's placement for free. **Only the root is mutable, and +it is 32 bytes.** That is the whole trick: concentrate all mutability into one +tiny signed value, then make that one value durable by other means. + +### Manifests are objects + +A manifest is stored via the ordinary object path at the `hot` tier +(3-copy — small, hot, cheap to repair; see CIP-001). It gets a manifest of its +own, which would recurse forever, so the base case: a manifest small enough to +fit in one block is replicated directly by hash with no manifest-of-manifest, +and its `n` locations are recorded in the parent snapshot. + +Large manifests are a real problem. CIP-002's open question notes a 1 TiB +object at 4 MiB blocks yields ~262k blocks and a manifest in the tens of MB. +Fix it two ways: + +1. **Scale block size with object size.** Target ≤4096 blocks per object: + `block_size = max(4 MiB, next_pow2(object_len / 4096))`. A 1 TiB object gets + 256 MiB blocks and a ~600 KiB manifest. Recorded per-object in the manifest, + so nothing is hard-coded. +2. **Chunk oversized manifests** into a two-level manifest as a normal object. + +### The snapshot + +A snapshot is an immutable, content-addressed map from name to object hash. In +this CIP it is a flat index; CIP-007 replaces the contents with an inode tree +without changing the mechanism. + +Serialised as a **HAMT** (hash array mapped trie) of fixed-size nodes, each +node stored as its own content-addressed block. This matters more than it +sounds: a snapshot must be cheap to *update*, not just to read. With a HAMT, +changing one entry rewrites only the ~log₃₂(N) nodes on the path to the root +and shares every other node with the previous snapshot. A million-entry volume +costs ~4 node writes per change instead of rewriting a million-entry file. + +Structural sharing also makes point-in-time snapshots nearly free, which is +where CIP-013's database backup story comes from. + +### The root pointer + +```json +{ + "volume": "vol_7f3a9c2e", + "sequence": 41207, + "snapshot": "blake3:...", + "parent": "blake3:...", + "written_at_ms": 1756512000000, + "writer_did": "did:coinpay:...", + "signature": "..." +} +``` + +`sequence` increments by exactly one per update. A reader that sees a gap knows +it is missing history; a writer that sees its expected sequence already taken +knows it lost a race (CIP-010). + +Durability for the root uses three mechanisms, because it is the one thing +whose loss is unrecoverable: + +1. **CoinPay anchor (authoritative).** The DID's registry entry holds the + current root. CoinPay is already the identity, payment, and reputation layer + under DIP-0007, so this adds no new central dependency — and DIP-0011's "no + central backend" already names CoinPay as a source of truth. +2. **Gossipsub announcement.** Roots publish to a `c0mpute/storage/roots/v1` + topic. Storage nodes holding the volume's shards cache the latest signed + root they've seen. Cheap, fast, self-healing, not authoritative. +3. **Local journal.** The writing client keeps every root it has written. + Enough to recover alone if CoinPay is unreachable at recovery time. + +A root is only advanced **after** the snapshot it names is fully placed. The +ordering is: write blocks → write manifests → write snapshot nodes → advance +root. A crash anywhere before the last step leaves the previous root valid and +some unreferenced garbage, which is the correct failure mode — never a root +pointing at data that isn't there. + +### Garbage collection + +Immutability plus mutable roots means orphans: superseded snapshot nodes, +manifests for deleted objects, blocks nobody references. + +- Roots retain their last `N` ancestors (default 32, configurable) as an undo + history. Anything reachable from a retained root is live. +- **Mark-and-sweep, per volume, client-driven.** The client walks reachable + hashes from retained roots and publishes a signed *keep-set digest*. Storage + nodes hold a shard while any keep-set references it, or until a grace period + (default 14 days) expires with no keep-set mentioning it. +- The grace period is what makes an offline client safe: a laptop that is shut + for a week does not lose its data. It also means deletion is not instant, and + billing must reflect held-not-referenced bytes — CIP-006's problem. +- Refcounting, per DIP-0012's open question, is *not* used: with content- + addressed dedup across volumes, refcounts require global coordination that + DIP-0011 rules out. Grace-period sweep is weaker but decentralised. + +### Recovery + +Given only the DID private key: + +``` +c0mpute storage recover --did did:coinpay:... --volume vol_7f3a9c2e +``` + +Reads the root from CoinPay, fetches the snapshot, walks manifests, verifies +shard availability, and reports what is intact, degraded, or lost — before +mounting anything. Losing every client machine costs nothing but a re-sync. + +## Acceptance criteria + +1. `c0mpute storage volume create` returns a volume id; `list` shows it with + its sequence and snapshot hash. +2. 1000 sequential object writes produce 1000 root updates with strictly + increasing sequence and no gaps. +3. Updating one entry in a 100k-entry volume writes fewer than 10 snapshot + nodes (proves structural sharing). +4. `kill -9` the client mid-write: the root still resolves to the previous + snapshot and every object it names reads back correctly. +5. Wipe the client's local state entirely; `recover` with only the DID key + reconstructs the full object list and reads every object. +6. A 1 TiB object produces a manifest under 1 MB (proves block-size scaling). +7. Deleting an object and running GC frees its shards after the grace period, + and not before. +8. A root signed by a different DID is rejected. + +## Risks + +- **CoinPay becomes a hard dependency for every write.** A root update per + write means CoinPay write latency is in the filesystem's critical path. + *Mitigation:* batch root updates — advance at most every `T` ms (default 500) + or every `N` operations, whichever first; `fsync` forces one immediately + (CIP-008). Gossip carries the root between anchors so readers aren't blocked. +- **CoinPay outage stalls durability.** *Mitigation:* keep writing to the local + journal and gossip; queue the anchor. Report degraded state in `doctor` + rather than failing writes. The data is safe; only the authoritative pointer + lags. +- **Grace-period GC lets a departed customer's data linger, billed.** + *Mitigation:* explicit `volume destroy` publishes a tombstone that skips the + grace period. Billing stops at tombstone, not at sweep. +- **HAMT is real work to get right.** *Mitigation:* it is a well-specified + structure with reference implementations; budget for property tests against a + naive map rather than inventing anything. + +## Estimate + +**3–4 weeks.** ~1 week HAMT with property tests, 1 week root pointer plus +CoinPay anchoring and gossip, 0.5 week manifest-as-object and block-size +scaling, 1 week GC and keep-sets, 0.5 week recovery command. + +## Open questions + +- Retained-root depth of 32: enough for a useful undo window, or should it be + time-based (e.g. 7 days of roots)? +- Should the keep-set digest be a Bloom filter to keep it small? False + positives retain garbage, which is safe; false negatives delete live data, + which is not — so the filter must be sized conservatively or made exact. +- Is one root per volume the right granularity, or should large volumes shard + the root by subtree to reduce write contention? CIP-010 may force this. diff --git a/docs/prds/005-repair-daemon.md b/docs/prds/005-repair-daemon.md new file mode 100644 index 0000000..b157485 --- /dev/null +++ b/docs/prds/005-repair-daemon.md @@ -0,0 +1,233 @@ +--- +cip: 005 +title: "Auto-repair daemon" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 (0012-storage-plugin.md) Phase 4 +depends-on: 003, 004 +blocks: +implementation: +estimate: "3–4 weeks" +--- + +## Summary + +Continuously detect blocks that have lost shards and regenerate them onto fresh +peers before loss becomes unrecoverable. CIP-001 established that our cost +advantage is bought by spending Storj's durability margin, which means repair +speed is not an operational nicety — it is the component that makes RS 10/14 a +defensible choice instead of a reckless one. + +## Motivation + +RS 10/14 tolerates 4 lost shards. At 5% monthly node churn, a block loses its +first shard within weeks and its fourth within a few months. Without repair, +every object in the network trends toward unrecoverable on a predictable +timetable. There is no version of this program where repair is optional or +deferred. + +Compare the two designs honestly, from CIP-001's table: + +| | c0mpute RS 10/14 | Storj RS 29/80 | +|---|---|---| +| Tolerates | 4 losses | 51 losses | +| Durability @ p=0.95 | 3.4 nines | 46.8 nines | +| Repair urgency | **hours** | weeks | +| Storage cost | 1.4x | 2.76x | + +We chose the left column. The bill for that choice is paid here. + +## Goals + +- Detect shard loss within one scan interval (target: 1 hour). +- Repair a degraded block to full `n` within 6 hours of detection. +- Never let repair traffic starve customer reads. +- Repair without needing the customer's client to be online. +- Prove repair happened, so providers can be paid and reputations adjusted. + +## Non-goals + +- Proving a node *currently holds* a shard it claims to (CIP-006 challenges). + This CIP trusts `Have` responses; a lying node is a CIP-006 problem. +- Repairing data whose root pointer is lost — unreachable is not degraded, it + is garbage, and CIP-004's GC handles it. + +## Design + +### Who repairs? + +Not the customer's client: a laptop that is closed for a week cannot be the +thing standing between the network and data loss. + +Repair is performed by **shard-holding nodes acting for the blocks they +already participate in.** For each block a node holds a shard of, it is a +candidate repairer. To avoid 14 nodes all repairing the same block +simultaneously, the repairer is deterministic: + +``` +repairer_for(block, round) = argmin over healthy holders h of + blake3(block_hash || round || h.peer_id) +``` + +Every holder computes the same answer without coordination. If the elected +repairer is itself gone, the next round elects someone else. This is rendezvous +hashing, and it needs no consensus — which matters under DIP-0011. + +### Detection + +Each storage node scans the blocks it participates in, on a rolling schedule +sized so every block is checked once per `scan_interval` (default 1 hour): + +1. Batch `Have` probes (CIP-003's batched existence probe) to the other `n-1` + holders. +2. Count healthy shards. +3. Classify: + +| Healthy shards (of 14) | State | Action | +|---|---|---| +| 14 | `healthy` | none | +| 12–13 | `degraded` | repair, normal priority | +| 11 | `urgent` | repair, high priority, preempt background work | +| ≤10 | `critical` | repair immediately; alert; block is one loss from death | +| <10 | `lost` | cannot repair; record and surface loudly | + +Thresholds are `(k, n)`-relative, not absolute: repair triggers at +`n - floor(parity/2)` healthy, i.e. as soon as half the parity budget is spent. +Waiting until `k+1` would be cheaper and is what Storj's margin lets them do; +we cannot afford it. + +### Repair + +To repair block `B` missing shards `{i, j}`: + +1. Acquire a repair lease on `B` via gossip (`c0mpute/storage/repair/v1`), + naming the repairer and an expiry. Duplicate work is wasteful, not + incorrect — the lease is an optimisation, and a lost lease race just means + two nodes repair the same block. +2. Fetch `k` shards from healthy holders, streaming (CIP-003). +3. RS-reconstruct, verify against the block hash in the manifest. +4. Regenerate only the missing shards `{i, j}` — not all `n`. +5. Select replacement peers under CIP-003's diversity constraints, **excluding + every peer already holding a shard of this block**. +6. `Put` the regenerated shards. +7. Publish a signed repair attestation (see below) and update `host_hint` in + the manifest, which requires advancing the volume root (CIP-004). + +Step 7 is the awkward one: repair mutates metadata the customer owns. The +repairer cannot sign the customer's root. Resolution: `host_hint` is +explicitly a *hint*, and the DHT is the source of truth for shard location +(CIP-003). Repair updates DHT provider records, which needs no customer +signature, and the stale hint is corrected opportunistically the next time the +customer's client writes. **Reads never depend on hints being fresh.** + +### Repair attestations + +```json +{ + "block": "blake3:...", + "object": "blake3:...", + "repairer": "did:coinpay:...", + "round": 41, + "shards_regenerated": [3, 11], + "sources": ["did:...", "..."], + "bytes_read": 4194304, + "completed_at_ms": 1756512000000, + "signature": "..." +} +``` + +Published to gossip and retained by holders. Three uses: proving repair for +provider reputation, attributing repair bandwidth (unpaid but measured, per +CIP-001), and detecting nodes that repeatedly fail to serve repair reads. + +### Not starving customers + +Repair is bulk background traffic competing with latency-sensitive reads on +consumer uplinks. Controls: + +- A per-node repair bandwidth budget, default **20% of measured uplink**, + configurable, enforced by a token bucket on repair streams. +- Repair reads are marked low-priority; a node under customer read load sheds + repair first. +- `urgent`/`critical` blocks bypass the budget — data loss beats latency. +- Global backpressure: if a node's repair queue exceeds a threshold, it stops + accepting *new* shard placements. A node that cannot keep its existing data + healthy has no business taking more. + +### The churn storm + +The failure mode that kills p2p storage networks: a large operator leaves, +mass repair starts, repair traffic saturates uplinks, healthy nodes time out +and are misclassified as failed, which triggers more repair. Reflexive +collapse. + +Defences: + +- **Distinguish unreachable from gone.** A shard is only presumed lost after + `grace_probes` (default 6) failures spread over `grace_window` (default 2 + hours). Brief outages must not trigger repair; most consumer nodes flap. +- **Announced departures drain gracefully.** A node running + `c0mpute storage retire` announces, keeps serving while its shards are + re-placed, and exits clean with reputation intact. Make the good path + attractive so it is the common one. +- **Network-wide repair rate limit.** If more than `X%` (default 5) of blocks + are simultaneously degraded, the network is in a storm: repair proceeds + strictly in priority order (`critical` first) at a capped global rate rather + than everyone repairing everything at once. +- **Never repair onto a node that is itself draining.** + +## Acceptance criteria + +1. On a 20-node testnet, killing 2 of 14 holders of a block results in a fully + repaired block within one scan interval, on fresh peers satisfying diversity. +2. Exactly one node performs the repair (rendezvous election verified in logs); + under an induced lease race, at most 2 do, and the result is still correct. +3. Repair regenerates only missing shards — the 12 survivors are byte-identical + before and after. +4. A node offline for 30 minutes and back does **not** trigger repair of its + shards. +5. Under a synthetic 100 Mbit uplink cap, customer read p99 rises less than 20% + while repair runs at its budget. +6. Retiring a node with `retire` re-places all its shards with zero degraded + blocks at any point. +7. Simulated 30%-of-network departure: no cascading failure; repairs complete + in priority order; no block reaches `lost`. +8. Repair attestations verify against the repairer's DID. + +## Risks + +- **Repair is unpaid, so nodes are incentivised to skip it.** A node that never + repairs saves bandwidth and loses nothing directly. *Mitigation:* attestations + feed reputation; low-reputation nodes stop receiving placements and therefore + stop earning. Make repair participation a condition of the storage role. + This deserves its own scrutiny in CIP-006 — it is the sharpest incentive + misalignment in the design. +- **Scan cost grows with data held.** A node with 10 TB and 4 MiB blocks tracks + ~2.5M blocks; probing all hourly is a lot of messages. *Mitigation:* batched + `Have` probes; scan at the *object* level with per-block detail only on + suspicion; scale block size with object size (CIP-002/004) which cuts block + count by orders of magnitude on large objects. +- **Repair storms hitting CoinPay.** *Mitigation:* repair touches DHT records, + not roots — deliberately, per the design above. +- **The 20% bandwidth budget is wrong for real uplinks.** *Mitigation:* it is + configurable; measure across a real fleet before defaulting anything. + +## Estimate + +**3–4 weeks.** ~0.5 week detection and classification, 0.5 week rendezvous +election and leases, 1 week the repair path, 0.5 week attestations, 1 week +bandwidth control and storm defences, 0.5 week the chaos test harness. + +## Open questions + +- Should repair be *paid* after all, funded from the storage margin, to fix the + incentive problem at its root rather than via reputation? CIP-001's cost model + says a paid repair egress at 5% churn costs $0.0014/GB-month against a + $0.0014/GB-month gross margin — so paying full rate is impossible, but paying + a fraction may be affordable and worth it. +- Is one hour the right scan interval? It sets worst-case exposure directly. +- Should `critical` blocks trigger a temporary tier upgrade (extra parity) + until the network is healthy again? diff --git a/docs/prds/006-challenges-metering-payouts.md b/docs/prds/006-challenges-metering-payouts.md new file mode 100644 index 0000000..936a0fa --- /dev/null +++ b/docs/prds/006-challenges-metering-payouts.md @@ -0,0 +1,240 @@ +--- +cip: 006 +title: "Storage challenges, metering, and provider payouts" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 (0012-storage-plugin.md) Phase 5, DIP-0007 (CoinPay DID) +depends-on: 003, 004 +blocks: +implementation: +estimate: "4–6 weeks" +--- + +## Summary + +Pay people for the disk they contribute, and make sure they are actually +contributing it. This is the half of the product that creates supply: without +payouts there are no providers, and without challenges the payouts fund fraud. + +## Motivation + +The whole design assumes a pool of nodes that hold shards honestly for money. +Neither the money nor the honesty exists yet. `c0mpute-verify` has the right +primitive already — `StorageChallenge` with byte-range proofs and a tested +`expected_response` — but nothing ever issues one, and no payout path exists +for storage at all. + +Getting this wrong in the obvious way is expensive: a node that stores nothing, +answers `Have` with "yes", and collects $0.0015/GB-month is strictly more +profitable than an honest one, right up until a customer tries to read. + +## Goals + +- Prove a node holds the bytes it claims, cheaply and continuously. +- Meter stored bytes and served bytes attributably. +- Pay providers monthly through CoinPay with an auditable statement. +- Bill customers for storage, egress, and operations. +- Make cheating cost more than it earns. + +## Non-goals + +- Staking, bonding, or slashing deposits. Reputation and payout forfeiture are + the levers; CIP-001 explicitly rules out collateral. +- Fiat payouts or tax handling — CoinPay's problem, not ours. +- Preventing Sybil identity creation (accepted residual risk, CIP-003). + +## Design + +### Challenges + +A challenge asks: *hash bytes `[offset, offset+len)` of shard `S`*. The +verifier knows the answer because it can compute it from any `k` shards of the +block; the holder can only answer by possessing the shard. + +```rust +pub struct StorageChallenge { + pub chunk_hash: c0mpute_proto::Hash, + pub offset: u32, + pub length: u32, +} +``` + +Already implemented and tested. What's missing is issuance, scheduling, +response verification, and consequences. + +**Who challenges.** Peer holders of the same block, on the same rendezvous +rotation as CIP-005's repair election but offset by round, so challenging and +repairing responsibilities spread evenly. The customer's client also challenges +opportunistically when online — but the network must not depend on it. + +**Rate.** Each shard is challenged on average once per 24 hours, with the +offset chosen from a per-round seed the holder cannot predict in advance. A +32 KiB range read plus a hash is negligible for an honest node and impossible +to fake without the data. + +**Precomputation attack.** A node that stores only *answers* to past challenges +rather than the shard defeats a fixed challenge set. Defence: the range is +derived from `blake3(shard_hash || round_seed || challenger_did)` where +`round_seed` comes from a recent gossip beacon, so the answer space is +unbounded and unpredictable. Storing enough precomputed answers costs more than +storing the shard. + +**Response deadline** 5 seconds. Late is a fail; a node that has the data but +cannot serve it inside 5s is not providing a usable service. + +**Consequences**, graded — most failures are innocent (reboots, flaky +connections), so the ladder starts gently: + +| Pass rate (30d) | Effect | +|---|---| +| ≥ 99% | Full payout, eligible for new placements | +| 95–99% | Full payout, no *new* placements until recovered | +| 80–95% | Payout scaled by pass rate; shards drain via repair | +| < 80% | Payout forfeit for the period; shards drained; role suspended | + +Failures feed `verification_pass_rate` in the existing +`c0mpute-verify::reputation` formula, which already weights it at 0.15 and +carries a 0.50 slash term. + +### Metering + +Two meters, both needing to be attributable and hard to inflate. + +**Stored bytes.** Sampled hourly: for each shard a node holds and has passed a +challenge on within the window, accrue `bytes × hours`. Accrual requires a +recent passing challenge, so claiming storage you don't have earns nothing. +Data held past deletion during CIP-004's GC grace period **is** billable to the +network but **not** to the customer — that is a real cost of the decentralised +GC design, absorbed by margin. It is small (14 days of a fraction of deleted +data) but it must be budgeted rather than discovered. + +**Served bytes.** The serving node reports bytes served, and the *receiving* +party countersigns a receipt. Neither side can inflate alone: + +```json +{ + "server_did": "did:coinpay:...", + "client_did": "did:coinpay:...", + "object": "blake3:...", + "bytes": 4194304, + "internal": true, + "served_at_ms": 1756512000000, + "server_sig": "...", "client_sig": "..." +} +``` + +`internal: true` means the reader was a c0mpute worker, which is billed to the +customer at $0 (CIP-001's compute-locality promise) but still **paid to the +provider** at the egress rate. That asymmetry is deliberate and is a genuine +cost centre: internal reads earn nothing and cost egress payouts. It is the +price of the differentiator, and it must be sized before launch — if internal +reads dominate, the $0-internal-egress promise needs a fair-use ceiling. + +Repair traffic generates receipts marked `repair: true`, which are **measured +but not paid** (CIP-001). Measuring them anyway is what makes CIP-005's +incentive problem visible. + +### Payouts + +Monthly, through CoinPay, to the provider's DID. + +``` +payout = stored_gb_month × $0.0015 × pass_rate_multiplier + + customer_egress_gb × $0.002 + - penalties +``` + +Statement available via `c0mpute storage earnings [--month YYYY-MM]`, itemised +by volume-agnostic aggregate (never revealing which customer's data a node +holds). Minimum payout threshold $5, rolling over below that. + +### Customer billing + +``` +bill = Σ tier_price × gb_month (CIP-001 tier prices) + + internet_egress_gb × $0.004 + + $0 for internal egress + + $0 for operations +``` + +Per-operation charges are deliberately zero — the pricing analysis in +`docs/storage-pricing-scenarios.md` already found they round to noise below +billions of requests, and "no per-request fees" is a cleaner promise than +matching R2's $0.36/M. + +Billing runs off the same signed receipts as payouts, so the two reconcile by +construction. Any gap between what customers are billed and what providers are +paid is margin, and it should be continuously observable rather than computed +at month end. + +### Fraud + +| Attack | Defence | +|---|---| +| Claim storage, store nothing | Challenges; accrual gated on passing | +| Store one copy, claim `n` shards under `n` identities | Diversity constraints make co-location detectable; challenge all `n` simultaneously and measure response correlation | +| Inflate served bytes | Countersigned receipts; the client won't sign what it didn't get | +| Collude: operator runs both server and client, signs fake receipts | Egress payout capped at a multiple of stored bytes; anomalous ratios flagged. **Not fully solved** — see risks | +| Store, pass challenges, refuse real reads | Read failures are reported and feed `job_completion_rate` | +| Precompute challenge answers | Unpredictable seeded ranges | + +## Acceptance criteria + +1. A node deleting a shard it claims fails its next challenge within 24h, and + accrual for that shard stops. +2. Challenge cost is under 0.1% of a node's bandwidth at 10 TB held. +3. A node with a precomputed answer table for 1000 past challenges still fails + new ones. +4. Served-byte receipts require both signatures; a single-signed receipt is + rejected by the billing reconciler. +5. `c0mpute storage earnings` matches an independent recomputation from raw + receipts to the cent. +6. A node at 97% pass rate keeps full payout but receives no new placements. +7. A node at 70% is drained, suspended, and forfeits the period. +8. Customer bill and provider payouts reconcile: `Σ bills − Σ payouts` equals + expected margin within rounding. +9. A brief outage (2h) does not measurably reduce a node's monthly payout. + +## Risks + +- **Self-dealing egress collusion is not fully solved.** An operator running a + node and a client can sign receipts for reads that never happened, minting + egress payouts. The cap on egress-to-stored ratio limits the damage but does + not eliminate it. *Mitigation:* cap, anomaly detection, and manual review + above a payout threshold. This is a known open problem in every p2p storage + network; do not claim it is solved. +- **Challenge traffic at scale.** Every shard, daily, network-wide, is a lot of + messages. *Mitigation:* batch challenges per peer-pair; challenge at block + granularity, sampling one shard per block per round. +- **Providers churn out when they see real earnings.** $0.0015/GB-month means a + fully-utilised 4 TB drive earns ~$6/month. That is idle-resource arbitrage, + not a business, and the messaging must say so plainly — a provider who + expected $50 leaves within a month and drives up churn, which CIP-005 pays + for. *Mitigation:* an earnings estimator in onboarding, before install. +- **CoinPay payout failures.** *Mitigation:* accrue locally, retry, expose + pending balance; never silently drop a period. +- **Repair remains unpaid and unrewarded.** Carried from CIP-005; the open + question about partial repair funding should be settled in this CIP since + this is where the money is defined. + +## Estimate + +**4–6 weeks.** ~1 week challenge issuance and verification, 1 week metering and +receipts, 1 week payout computation and CoinPay integration, 1 week customer +billing and reconciliation, 0.5 week the earnings CLI, 1 week fraud controls and +adversarial tests. + +## Open questions + +- Should repair egress be partially paid (e.g. 25% of the egress rate) to fix + CIP-005's incentive gap? Costed at 5% churn this is ~$0.00035/GB-month against + a $0.0014 gross margin — affordable at a quarter rate, not at full. +- Payout cadence: monthly is simple, but weekly would reduce provider anxiety + early on. CoinPay transaction costs decide this. +- Does the $0-internal-egress promise need a fair-use ceiling? Provider egress + payouts on internal reads have no offsetting revenue. +- Minimum challenge sample: one shard per block per round, or every shard? + Cheaper sampling means a dishonest node survives longer. diff --git a/docs/prds/007-c0mputefs-filesystem.md b/docs/prds/007-c0mputefs-filesystem.md new file mode 100644 index 0000000..fc9859f --- /dev/null +++ b/docs/prds/007-c0mputefs-filesystem.md @@ -0,0 +1,260 @@ +--- +cip: 007 +title: "c0mputefs: mutable filesystem over immutable content" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 — supersedes its "filesystem-style mutable objects" out-of-scope line +depends-on: 004 +blocks: 008, 009, 011 +implementation: +estimate: "4–6 weeks" +--- + +## Summary + +Build the inode and directory layer that turns content-addressed immutable +blocks into a POSIX filesystem with paths, permissions, and mutation. This is +the conceptual centre of the read/write product and the piece that does not +exist in any form today. + +## A scope change this CIP makes explicit + +DIP-0012 currently lists under **Out of scope**: + +> Filesystem-style mutable objects. Content-addressed, immutable. + +The decision to ship a read/write mount overrides that line. This is a genuine +reversal, not an interpretation, and it should land as a short DIP amendment +alongside this CIP rather than being buried in a PRD. The reasoning is +recorded here so the amendment can cite it: + +- Content-addressed immutability remains true of **everything below the root + pointer**. Blocks, manifests, and snapshot nodes are still immutable. +- Mutability is confined to advancing one signed 32-byte pointer per volume + (CIP-004). The storage layer does not become mutable; a naming layer is added + on top of it. +- So the original property that made the design tractable is preserved. What + changes is that we now also ship the mutable index that customers were + otherwise going to have to build themselves. + +Worth stating plainly: a POSIX mount is a materially harder product than an +object store, and most of the difficulty lands in CIP-008 and CIP-010 rather +than here. This CIP is the tractable part. + +## Goals + +- Full POSIX metadata: inodes, modes, uid/gid, timestamps, link counts. +- Directories with atomic `rename(2)`, including the overwrite case. +- Files with random-access read and write, `truncate`, sparse regions. +- Hard links and symlinks. +- Metadata operations that complete in microseconds, not network round trips. +- Efficient small files — the pathological case for erasure-coded storage. + +## Non-goals + +- The write path's durability, journaling, and crash consistency (CIP-008). +- FUSE bindings and the CLI (CIP-009). +- Concurrent writers across mounts (CIP-010). +- Encryption (CIP-011). +- `O_DIRECT`, `mmap` shared-writable, mandatory locking. See Out of scope. + +## Design + +### Layering + +``` + VFS / FUSE (CIP-009) + │ + c0mputefs inodes, dentries, permissions, path resolution ← this CIP + │ + Volume root pointer + HAMT snapshot (CIP-004) + │ + Objects manifests, blocks, RS shards (CIP-002/003) +``` + +New crate: `node/crates/c0mpute-fs`. It depends on `c0mpute-store` and the +volume layer, and knows nothing about FUSE — which keeps it testable without a +mount and reusable by the S3 gateway (CIP-012) and by CIP-013's backup tooling. + +### Inodes + +```rust +pub struct Inode { + pub ino: u64, + pub kind: FileKind, // File | Dir | Symlink + pub mode: u32, // POSIX permission bits + pub uid: u32, pub gid: u32, + pub nlink: u32, + pub size: u64, + pub atime: u64, pub mtime: u64, pub ctime: u64, + pub content: Content, + pub xattrs: BTreeMap>, +} + +pub enum Content { + /// Small files live inline in the inode. No manifest, no shards. + Inline(Vec), + /// Larger files reference an extent tree of content-addressed blocks. + Extents(ExtentTree), + /// Directory entries: name -> ino, as a HAMT for large directories. + Dir(HamtRef), + /// Symlink target. + Link(String), +} +``` + +The inode table is itself a HAMT keyed by `ino`, stored in the volume snapshot +(CIP-004). Updating one inode rewrites ~4 HAMT nodes, not the table. + +### Inline small files — the small-file problem, solved by avoiding it + +A 2 KiB file under RS 10/14 becomes 14 shards of ~200 bytes on 14 different +machines, plus a manifest larger than the file. It is absurd, and it is the +single most common file size on a real filesystem. + +Files at or below `inline_max` (default **64 KiB**) are stored **directly in +the inode**, which lives in the snapshot HAMT, which is stored at the `hot` +tier (3-copy). Consequences: + +- No manifest, no erasure coding, no 14-way fan-out for small files. +- Reads come from the already-cached snapshot — often zero network round trips. +- Replication's 1x repair amplification applies, which is what CIP-001 said + replication was for. + +Above `inline_max`, content moves to an extent tree. Crossing the threshold in +either direction is a normal rewrite. + +### Extent trees + +```rust +pub struct ExtentTree { pub extents: Vec } + +pub struct Extent { + pub file_offset: u64, + pub len: u64, + pub content: ExtentContent, +} + +pub enum ExtentContent { + Block { hash: Hash, block_offset: u32 }, // content-addressed block + Zero, // sparse hole, stores nothing +} +``` + +An extent maps a byte range of the file onto a range of an immutable block. +Overwriting the middle of a file does not rewrite the file: it writes one new +block and splices three extents (prefix, new, suffix). `truncate` drops or +splits extents. Sparse files cost nothing. + +This is why CIP-002's `get_range` is a hard dependency — extents are useless if +reading 4 KiB requires reconstructing a whole object. + +### Content-defined chunking + +CIP-002 deferred this and the write path now forces the question. With fixed +block boundaries, inserting one byte at the start of a file re-encodes every +subsequent block. With **content-defined chunking** (rolling hash, FastCDC — +target 4 MiB, min 1 MiB, max 16 MiB) boundaries follow content, so an insert +changes one chunk and the rest dedup against what is already stored. + +For an append-only workload — logs, datasets, database WAL — this is the +difference between rewriting the file and appending a chunk. Adopt FastCDC for +extent content. Fixed blocks remain available for objects written through the +plain object API (CIP-002), which has no insert semantics. + +### Directories and rename + +Directory entries are a HAMT from name to `(ino, kind)`, so a directory with a +million entries costs `log₃₂(N)` node writes per change and `readdir` streams +without loading it all. + +`rename(2)` must be atomic, including "overwrite existing target". Since every +mutation produces a new snapshot and the volume advances one root pointer +atomically, **atomicity is inherited**: build a new snapshot with the entry +removed from source, added to target, target's old inode's `nlink` decremented, +then advance the root once. There is no window in which both or neither exists. +Getting rename atomicity nearly free is the main payoff of the CIP-004 design. + +### Path resolution and caching + +Resolution walks the dentry HAMT per component. Every layer is content- +addressed and immutable, so **caching is trivially safe**: a node's contents can +never change under a given hash. Cache HAMT nodes and inodes by hash in a +bounded LRU (default 256 MB); invalidation happens only when the root advances, +and even then only along the changed path. + +Metadata operations hit this cache, not the network. `stat` on a hot path is a +hash lookup in memory. This is what makes the mount feel like a filesystem +rather than a network protocol, and it is why the design puts metadata in a +snapshot rather than fetching per-inode records. + +### Permissions + +Standard POSIX mode/uid/gid checks, enforced in the FUSE layer (CIP-009) with +`default_permissions`. uid/gid are stored as written; there is no network-wide +identity mapping, so a volume mounted on two machines with different uid +namespaces sees different owners. Documented, with a `-o uid=,gid=` squash +option for the common single-user case. + +## Acceptance criteria + +1. `pjdfstest` POSIX conformance suite passes for `chmod`, `chown`, `link`, + `symlink`, `mkdir`, `rmdir`, `rename`, `truncate`, `unlink` (write-path + durability tests deferred to CIP-008). +2. A 4 KiB file writes zero shards and zero manifests; it round-trips from the + snapshot alone. +3. Writing 1 byte at offset 0 of a 1 GiB file uploads one chunk, not 1 GiB + (asserted via a bytes-uploaded counter). +4. Appending 1 MiB to a 10 GiB file re-uploads under 20 MiB (FastCDC working). +5. `rename` over an existing file is atomic under `kill -9` in a loop: the + target is always either the old or the new inode, never missing. +6. A directory with 1M entries: `readdir` streams in constant memory; creating + one more entry writes fewer than 10 HAMT nodes. +7. `stat` on a cached path completes in under 100 µs with no network I/O. +8. A sparse 1 TiB file with 1 MiB written consumes ~1 MiB. +9. `cargo test -p c0mpute-fs` covers extent splice, truncate, and inline + promotion/demotion across the threshold. + +## Risks + +- **POSIX is a large surface with sharp edges.** `rename` over a non-empty + directory, `unlink` of an open file (must stay readable until close), `nlink` + accounting for hard links. *Mitigation:* `pjdfstest` from the first week, not + at the end; treat it as the definition of done. +- **Unlink-while-open.** POSIX requires the inode to survive until the last + descriptor closes. Needs an orphan list in the volume so a crash mid-open + doesn't leak. *Mitigation:* explicit orphan inode set in the snapshot, swept + at mount. +- **Cache coherence across mounts.** Safe here only because this CIP assumes a + single writer; CIP-010 is where this gets hard. +- **FastCDC adds CPU on the write path.** ~1 GB/s/core for a rolling hash is + fine for network-bound writes but not free on a node also running inference. + *Mitigation:* benchmark; make chunking parameters and enablement per-volume. + +## Estimate + +**4–6 weeks.** ~1 week inode model and HAMT integration, 1 week extent trees +and splice logic, 0.5 week directories and rename, 1 week FastCDC and the +inline/extent threshold, 0.5 week caching, 1–2 weeks `pjdfstest` conformance +(assume this expands). + +## Out of scope + +- `O_DIRECT` — no meaningful semantics over a caching network filesystem. +- Shared writable `mmap`. Private read-only `mmap` works. +- Mandatory locking, `flock` across mounts (CIP-010 covers advisory leases). +- Quotas per directory. Volume-level accounting only. +- Case-insensitive lookup. + +## Open questions + +- `inline_max` of 64 KiB: bigger inlines make snapshots larger, which are + rewritten on every metadata change. 16 KiB may reconcile better with the HAMT + node size. Needs measurement on a real corpus. +- Should `ino` be a hash of the path (stable across snapshots, breaks on + rename) or a counter (stable across rename, needs allocation state)? Counter + is proposed; POSIX callers assume inode stability across rename. +- Do xattrs belong inline in the inode, or as a separate object above a size? diff --git a/docs/prds/008-write-path-consistency.md b/docs/prds/008-write-path-consistency.md new file mode 100644 index 0000000..0fdd495 --- /dev/null +++ b/docs/prds/008-write-path-consistency.md @@ -0,0 +1,227 @@ +--- +cip: 008 +title: "Write path: chunking, journal, and crash consistency" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 +depends-on: 007 +blocks: 009, 013 +implementation: +estimate: "4–5 weeks" +--- + +## Summary + +Make writes fast enough to be usable and honest enough to be trusted. A local +write-back journal on the client's own disk absorbs writes at local-disk speed; +a background uploader turns journalled data into placed shards; and `fsync(2)` +means something specific and documented rather than something hopeful. + +This is the CIP where the read/write product is won or lost. Everything else is +plumbing by comparison. + +## Motivation + +Applications assume filesystem writes are fast and that `fsync` means durable. +Neither is naturally true over a network of consumer nodes with 200–500 ms +latencies: + +- A naive synchronous write costs a full RS encode plus `n` peer round trips — + hundreds of milliseconds for a 4 KiB write. `tar -x` would take days. +- If `fsync` returns before data is placed, a crash loses acknowledged writes + and every durability claim in the product is false. + +The reconciliation is a local journal: acknowledge into durable local storage, +upload asynchronously, and define `fsync` in terms of what has actually +happened rather than what we would like to have happened. + +## Goals + +- Buffered writes complete at local-disk latency. +- `fsync` provides a precise, documented durability guarantee. +- A client crash never loses data that `fsync` acknowledged. +- A client crash never leaves the volume inconsistent. +- Bounded, observable divergence between local and network state. + +## Non-goals + +- Multi-writer coordination (CIP-010). +- Making `fsync` fast. It is a network round trip and will cost one. + +## Design + +### The journal + +Every mount owns a local journal on ordinary local disk (default +`~/data/c0mpute/journal/`), which should be the fastest device +available — the mount's write performance is the journal's write performance. + +``` +journal/ + wal/000042.log append-only intent records + staging/ chunk payloads awaiting upload + state.db upload queue, dirty inodes, sequence watermarks +``` + +Records are append-only and CRC'd: + +```rust +enum JournalRecord { + ChunkStaged { hash: Hash, len: u32 }, + ChunkPlaced { hash: Hash, hosts: Vec }, + InodeUpdate { ino: u64, inode: Inode }, + DirUpdate { parent: u64, name: String, entry: Option<(u64, FileKind)> }, + RootAdvanced { sequence: u64, snapshot: Hash }, + Barrier { id: u64 }, +} +``` + +### Write path + +``` +write(2) + └─► write into page cache, mark inode dirty ~µs, returns + └─► chunker seals a chunk (FastCDC boundary or flush) + └─► append ChunkStaged + payload to journal ~local disk + └─► uploader: RS-encode, place n shards (CIP-003) + └─► append ChunkPlaced + └─► root advance batches dirty inodes (CIP-004) + └─► append RootAdvanced, free staged payload +``` + +Writes return at the second step. Everything after is asynchronous, rate-limited +and observable. + +### What fsync means + +Three modes, chosen per-mount, because there is no single right answer and +pretending otherwise is how people lose data: + +| Mode | `fsync` returns when | Survives | Latency | +|---|---|---|---| +| `local` | journal `fdatasync`'d to local disk | client crash / power loss | ~1 ms | +| `network` (**default**) | shards placed at write-ack quorum (CIP-003: `k + parity/2`) **and** root advanced | total client loss | ~200–800 ms | +| `paranoid` | all `n` shards placed and CoinPay anchor confirmed | client loss + slow peers | ~1–3 s | + +**`network` is the default** because it is the only mode matching what an +application means by "durable" in a distributed system: the data survives the +machine that wrote it. `local` is offered for workloads that fsync constantly +and treat the network as a replica — it is honest, but it does not survive +losing the client, and the CLI says so at mount time. + +The mode is reported in `statfs` and by `c0mpute storage status`, so a database +can be configured against a known guarantee rather than a guess. CIP-013 depends +entirely on this table being accurate. + +### Crash consistency + +The invariant: **the volume root never references data that is not placed.** +CIP-004 already orders block → manifest → snapshot → root. The journal +preserves that ordering across a crash. + +On mount after an unclean shutdown: + +1. Scan the WAL forward from the last `RootAdvanced`. +2. Re-stage any `ChunkStaged` without a matching `ChunkPlaced`; re-queue upload. +3. Rebuild dirty inode state from `InodeUpdate`/`DirUpdate` after the watermark. +4. Compare local root sequence with the volume's anchored root: + - local ahead → replay to catch the network up (normal case: crash after + journalling, before anchoring). + - local behind → another writer advanced it; CIP-010 conflict path. + - equal → clean. +5. Sweep orphan inodes (CIP-007's unlink-while-open list). + +Recovery is bounded by journal size, not volume size. A 10 TB volume with a +2 GB journal recovers in seconds. + +### Bounding divergence + +An unbounded journal is a lie: it looks like a working filesystem while +accumulating data that exists on exactly one disk — precisely the failure the +product claims to prevent. + +- `journal_max_bytes` (default 8 GB) and `journal_max_lag_secs` (default 300). +- Approaching either, writes are **throttled** to the measured upload rate. +- At the limit, writes block (`EAGAIN` for non-blocking) rather than silently + buffering more. +- `c0mpute storage status` always shows journal depth, upload rate, and + estimated drain time. Users must be able to see how far behind they are. + +Backpressure over silent buffering is a deliberate choice: a slow filesystem is +an annoyance, a filesystem that loses a day's work on a laptop failure is a +product failure. + +### Read-your-writes + +Reads consult the journal first, then the network. A just-written chunk still +staging is served from local staging, so an application never sees a write it +made disappear. Consistency within a mount is total, regardless of upload lag. + +### Batching root advances + +One root advance per write would mean one CoinPay anchor per write. Batch: +advance at most every `root_batch_ms` (default 500) or every 1000 dirty inodes, +whichever first. `fsync` in `network`/`paranoid` mode forces an immediate +advance. Between advances the journal is authoritative and safe. + +## Acceptance criteria + +1. `dd if=/dev/zero of=/f bs=4k count=100000` sustains within 20% of the + same write to the journal's local device. +2. `fsync` in `network` mode does not return until shards are at quorum: kill + the client immediately after `fsync` returns, mount elsewhere, and the data + is present. 1000 iterations, zero losses. +3. `fsync` in `local` mode survives `kill -9` but is documented as not + surviving client loss; a test asserts this distinction explicitly. +4. Power-loss simulation (`dm-flakey` or equivalent) 1000 times: the volume is + always consistent; no root references a missing block. +5. `tar -xf linux-6.x.tar.xz` into the mount completes within 3x of the same + extraction on local disk. +6. Filling the journal throttles rather than erroring, and blocks rather than + buffering unbounded; `status` shows lag throughout. +7. Recovery from a 2 GB dirty journal completes in under 30 seconds. +8. Read-your-writes holds under a concurrent write/read loop with uploads + artificially stalled. +9. `fio` random 4 KiB writes with `fsync=1` in `network` mode reports latency + consistent with the documented table (not silently faster, which would mean + fsync is lying). + +## Risks + +- **Journal device failure loses un-uploaded data.** The window is real and + proportional to lag. *Mitigation:* bound the lag; document it; offer + `journal_mirror` to a second local device for critical mounts. +- **`fsync` latency makes some applications unusable.** SQLite in rollback mode + fsyncs per transaction; at 500 ms that is 2 tps. *Mitigation:* this is + physics, not a bug — CIP-013 documents which database configurations are + viable rather than pretending all are. +- **FastCDC plus journaling is CPU-heavy on a worker also running inference.** + *Mitigation:* cgroup the mount's CPU; make chunking parameters tunable. +- **Batched root advances widen the crash window in `local` mode.** Up to + `root_batch_ms` of metadata lives only in the journal. *Mitigation:* correct + by design (journal is durable and replayed), but it must be understood when + reasoning about `local` mode. +- **Divergence throttling will be perceived as the product being slow.** + *Mitigation:* surface *why* in `status` and in the throttle message. A slow + filesystem that explains itself is tolerable. + +## Estimate + +**4–5 weeks.** ~1 week journal format and replay, 1 week the async uploader and +backpressure, 0.5 week the three fsync modes, 0.5 week read-your-writes and +staging reads, 1 week crash-consistency test harness (`dm-flakey`), 1 week +performance work against the acceptance targets. + +## Open questions + +- Should `network` fsync wait for the root anchor, or only for shard quorum + plus a journalled root? Waiting on CoinPay adds latency for a guarantee the + gossip layer largely provides. +- Is 8 GB the right default journal cap? It should probably be a fraction of + free space rather than an absolute. +- Should `local` mode even be offered, given how easy it is to misread? It is + genuinely right for scratch space on a worker, but it is also the mode that + will be blamed when data is lost. diff --git a/docs/prds/009-mount-cli.md b/docs/prds/009-mount-cli.md new file mode 100644 index 0000000..87dc880 --- /dev/null +++ b/docs/prds/009-mount-cli.md @@ -0,0 +1,215 @@ +--- +cip: 009 +title: "`c0mpute storage` CLI and the FUSE mount" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 (`plugins/storage/module.toml` declares `cli = "c0mpute storage"`), DIP-0002 CLI structure +depends-on: 008 +blocks: 010, 013 +implementation: +estimate: "3–4 weeks" +--- + +## Summary + +The product surface. A `c0mpute storage` command group, and a FUSE filesystem +so a volume appears at a path where `ls`, `cp`, `rsync`, `git`, and every other +ordinary tool just work. + +## Motivation + +`plugins/storage/module.toml` already declares `cli = "c0mpute storage"` and +workloads `storage.put`, `storage.get`, `storage.repair`. None of it exists: +the CLI has a `--storage` flag for a directory path and a `Role::Storage` +variant, and that is all. There is no mount code of any kind in the tree. + +Everything CIPs 002–008 build is unreachable without this. + +## Goals + +- `c0mpute storage mount ` produces a working POSIX mount. +- Standard tools work unmodified. +- `/etc/fstab` and systemd automount support. +- A complete non-mount CLI for volumes, objects, providers, and diagnostics. +- Clear reporting of the state a distributed filesystem can be in. + +## Non-goals + +- Multi-writer safety (CIP-010). v1 mounts take an exclusive volume lease. +- Windows. Linux and macOS only, matching `module.toml`. +- NFS/SMB re-export (see Open questions). + +## Design + +### CLI surface + +Following DIP-0002's nesting, under `c0mpute storage`: + +``` +VOLUMES + volume create [--tier standard|hot|critical] [--quota GB] + volume list [--json] + volume info + volume destroy [--yes] + volume snapshot [--label ] + volume rollback --to + +MOUNTING + mount [-o opt,opt...] + umount + mounts + +OBJECTS (no mount required; the CIP-002 API) + put [--tier T] -> object hash + get [-o out] + ls [path] + rm + +PROVIDING + provide --disk [--path ] opt into the storage role + provide status + earnings [--month YYYY-MM] [--json] + retire [--drain-timeout ] graceful exit (CIP-005) + +DIAGNOSTICS + status [] journal lag, upload rate, degraded blocks + health per-object shard health + repair [--object H] force a repair pass + recover --did D --volume V rebuild from the DID key alone (CIP-004) + verify re-hash everything reachable; report mismatches +``` + +`provide` is the entry point for the supply side and should be the shortest +path in the product: one command, a disk budget, and an earnings estimate shown +before anything is committed (CIP-006's risk about disappointed providers). + +### Mount options + +``` +c0mpute storage mount vol_7f3a /mnt/data -o fsync=network,cache=256M +``` + +| Option | Default | Meaning | +|---|---|---| +| `fsync=local\|network\|paranoid` | `network` | CIP-008's durability mode | +| `cache=` | 256M | Metadata + block cache | +| `journal=` | `~/data/c0mpute/journal` | Put this on the fastest disk | +| `journal_max=` | 8G | Backpressure threshold | +| `ro` | off | Read-only; takes a shared lease, no journal | +| `uid=,gid=` | as-stored | Squash ownership for single-user mounts | +| `allow_other` | off | Standard FUSE semantics | +| `tier=` | volume default | Tier for newly written data | + +### fstab and systemd + +A `mount.c0mpute` helper installed in `/sbin` makes the ordinary syntax work: + +```fstab +vol_7f3a /mnt/data c0mpute _netdev,fsync=network,x-systemd.automount 0 0 +``` + +`_netdev` is essential — it orders the mount after the network, and without it +boot hangs. The installer adds the helper and a systemd unit template; both are +covered by acceptance tests, because an fstab entry that hangs boot is a much +worse bug than a mount that fails. + +### FUSE implementation + +The `fuser` crate, with `c0mpute-fs` (CIP-007) behind it. The FUSE layer is +deliberately thin — translation and permission checks only, with no filesystem +logic — so the semantics stay testable without a mount. + +- Multithreaded FUSE session; blocking network work off the reply threads. +- `default_permissions` so the kernel enforces mode bits. +- Kernel page cache enabled for reads; writeback cache **disabled**, since + CIP-008's journal is the write buffer and two layers of write buffering make + `fsync` semantics unanalysable. +- `readdirplus` to collapse `readdir` + `stat` storms (this is most of what + makes `ls -l` on a big directory feel bad). +- `statfs` reports quota as total, journal lag in `f_bavail` pressure, and the + fsync mode in the filesystem subtype so `mount` output shows it. + +**macOS:** `fuser` needs macFUSE, which requires a kernel extension and is +increasingly awkward. Ship Linux first-class; macOS via macFUSE if present, +with FUSE-T (NFS-loopback, no kext) as the documented fallback. Do not block +the Linux release on macOS. + +### Reporting distributed state + +A network filesystem has states a local one does not, and hiding them is how +users get hurt: + +``` +$ c0mpute storage status vol_7f3a +volume vol_7f3a (standard, RS 10/14) +mounted /mnt/data fsync=network +sequence 41207 anchored 3s ago +journal 412 MB pending, 38 files, draining ~14 MB/s (ETA 29s) +objects 18,432 total + 18,401 healthy + 29 degraded (repair queued) + 2 urgent (repair in progress) + 0 lost +network 47 storage peers, 12 ASNs +``` + +`degraded` and `urgent` are normal steady-state on a churning network and +should not read as alarming. `lost` is the one that matters, and it is called +out separately by `health` with the affected paths — a user needs to know +*which files*, not just a count. + +## Acceptance criteria + +1. `mount`, then `cp -a /usr/share/doc /`, then `diff -r` reports no + differences. +2. `git clone` a repo into the mount, `git status` clean, `git gc` succeeds. +3. `rsync -a --delete` in both directions converges with no errors. +4. `tar -xf` the kernel source and rebuild the file list identically. +5. An fstab entry with `_netdev` mounts on boot and, when the network is + unavailable, **fails without hanging boot**. +6. `umount` flushes the journal and releases the lease; a subsequent mount + elsewhere sees all data. +7. `ls -l` on a 10k-entry directory completes in under 1 second warm. +8. Killing the FUSE process leaves no stale mount; the next mount recovers per + CIP-008. +9. `c0mpute storage provide --disk 500` onboards a node, and `earnings` shows + an accrual within one billing interval. +10. `status` output above is accurate against an induced degraded state. +11. `volume rollback` restores a prior sequence and the mount reflects it. + +## Risks + +- **FUSE performance ceiling.** Context switches per operation put a floor on + latency; metadata-heavy workloads feel slower than local disk regardless of + our caching. *Mitigation:* `readdirplus`, aggressive metadata caching, honest + benchmarks in the docs. Consider `io_uring`-based passthrough later. +- **Stale mounts after a crash.** The classic FUSE annoyance — a `d` + directory that `ls` hangs on. *Mitigation:* explicit `fusermount3 -u` + handling on startup, a `mounts --clean` subcommand, and never leaving the + session without unmounting. +- **macFUSE friction on macOS.** *Mitigation:* FUSE-T fallback, and don't gate + the Linux launch. +- **`_netdev` omitted in a hand-written fstab hangs boot.** *Mitigation:* + `mount.c0mpute` refuses to mount at boot-time without `_netdev` and logs why; + the docs lead with the correct line. +- **Users will mount over an existing non-empty directory and lose sight of the + data underneath.** *Mitigation:* warn unless `--force`. + +## Estimate + +**3–4 weeks.** ~1.5 weeks the FUSE layer and its operation surface, 1 week the +CLI command group, 0.5 week fstab/systemd integration plus the mount helper, +0.5 week status/health reporting, 0.5 week macOS. + +## Open questions + +- NFS re-export (`fuser`'s `--export` / kernel NFS over FUSE) would give + Windows and appliance access for free. Worth it in v1, or does it multiply + the consistency surface before CIP-010 is settled? +- Should `mount` daemonise by default or run in the foreground? Foreground is + better for systemd; daemonising is what people expect interactively. +- Does `volume rollback` need to be blocked while mounted, or can it be applied + live with a cache invalidation? diff --git a/docs/prds/010-leases-multi-mount.md b/docs/prds/010-leases-multi-mount.md new file mode 100644 index 0000000..5ca615a --- /dev/null +++ b/docs/prds/010-leases-multi-mount.md @@ -0,0 +1,219 @@ +--- +cip: 010 +title: "Single-writer leases and multi-mount coherence" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012, DIP-0011 (no central backend) +depends-on: 009 +blocks: 013 +implementation: +estimate: "3–4 weeks" +--- + +## Summary + +Define what happens when the same volume is mounted in more than one place — +which is the first thing every user will try. v1 enforces **one writer, many +readers** with an explicit, expiring, DID-signed lease, and makes the failure +modes visible instead of silently corrupting data. + +## Motivation + +CIPs 004–009 all assume a single writer. That assumption is load-bearing: +without it, two mounts advancing the same root pointer produce lost updates, +and two mounts caching the same inodes produce stale reads. + +The user-visible reality is that people mount things twice — on a laptop and a +worker, on two workers in a job, or by accident after a crash left a lease +behind. The choice is between detecting that and handling it, or not detecting +it and losing data. Everything below follows from picking the first. + +Being explicit about the limit is also the honest thing to ship. "One writer at +a time, enforced" is a real product. "Multi-writer" that silently loses writes +is not. + +## Goals + +- At most one writer per volume at any time, enforced not merely documented. +- Readers see a consistent, bounded-staleness view while a writer is active. +- A crashed writer's lease expires and recovers without human intervention. +- Conflicts are detected and reported, never silently resolved by overwriting. +- No central lock service (DIP-0011). + +## Non-goals + +- True concurrent multi-writer with merge. Deferred; see Future work. +- Byzantine writers. A writer with the volume's key can corrupt its own volume; + this is an authorisation question, not a concurrency one. + +## Design + +### The write lease + +```json +{ + "volume": "vol_7f3a9c2e", + "holder_did": "did:coinpay:...", + "holder_mount": "hostname:/mnt/data", + "acquired_at_ms": 1756512000000, + "expires_at_ms": 1756512060000, + "epoch": 17, + "signature": "..." +} +``` + +- **60-second term, renewed every 20 seconds.** Two missed renewals expire it. +- `epoch` increments on every acquisition. A root update signed under an old + epoch is rejected — this is the actual enforcement, and it is what makes the + lease more than an advisory flag. +- Stored alongside the root pointer in the CoinPay registry (authoritative) and + gossiped on `c0mpute/storage/leases/v1` (fast path). + +Acquisition: read the current lease. If absent or expired, write a new one at +`epoch+1` via CoinPay's compare-and-set on the registry entry. CoinPay is +already the authority for the root pointer (CIP-004), so this introduces no new +central dependency — it reuses the one DIP-0011 already sanctions. + +If held and unexpired, the mount fails with a message naming the holder: + +``` +$ c0mpute storage mount vol_7f3a /mnt/data +error: volume vol_7f3a is mounted read-write by worker-3:/mnt/data + lease expires in 41s (renewing) +hint: mount read-only with -o ro, or use --steal if that mount is dead +``` + +### Read-only mounts + +Readers take no lease and are unlimited. A reader polls the root pointer +(gossip, falling back to CoinPay) every `root_poll_ms` (default 2000) and +advances its view atomically when the sequence increases. + +Because every layer below the root is immutable and content-addressed, a reader +holding a snapshot hash has a **consistent point-in-time view** for free. There +is no torn state: it sees snapshot `N` entirely, then snapshot `N+1` entirely. +Staleness is bounded by the poll interval and is reported by `statfs`. + +This is the strongest property the CIP-004 design gives us, and it means "many +readers" costs almost nothing to support correctly. + +### Stealing a dead writer's lease + +The common case: a laptop crashed while holding a lease. + +``` +c0mpute storage mount vol_7f3a /mnt/data --steal +``` + +Permitted only when the lease has expired. It bumps the epoch, which fences the +old writer permanently — if that machine wakes up, its next root update is +rejected on epoch and it enters the recovery path below rather than corrupting +anything. + +`--steal` never applies to an unexpired lease. Waiting 60 seconds is the price +of not having a distributed consensus protocol, and it is the right trade. + +### The fenced writer + +A writer whose lease expired (long GC pause, network partition, laptop sleep) +may hold journalled, un-uploaded writes. On rejection it must not discard them. + +1. Stop accepting new writes; the mount goes read-only immediately. +2. Report loudly via `status` and a `dmesg`-visible FUSE error. +3. Preserve the journal, and export the divergent state: + +``` +$ c0mpute storage status vol_7f3a +FENCED: lease lost at epoch 17; volume now at epoch 18 (worker-3) + 412 MB / 38 files written locally after divergence + recover with: c0mpute storage export-divergent vol_7f3a ./recovered +``` + +4. `export-divergent` writes the un-uploaded files to a local directory as + ordinary files, so nothing is lost even though it cannot be merged. + +Deliberately **no automatic merge.** Two divergent filesystem trees cannot be +merged safely without application knowledge — that is exactly the mistake that +makes distributed filesystems infamous. Surface it, preserve it, let a human +decide. + +### Coherence for readers during writes + +A reader advancing from snapshot `N` to `N+1` invalidates only cache entries +whose hash changed, which the HAMT makes cheap to compute: walk the two roots +and diff, pruning wherever the node hashes match. A typical advance touches a +handful of paths. + +Open file handles on a reader keep their inode-at-open, matching NFS +close-to-open semantics. An application wanting the new version reopens. This +is well-trodden behaviour that users already understand from NFS. + +### Advisory locking + +`flock(2)` and `fcntl` locks are honoured **within a single mount** by the +local FUSE layer. Across mounts they are not, because there is only ever one +writer — cross-mount write locks would be meaningless. + +`statfs` reports `single_writer` so applications can detect the model. CIP-013 +depends on this: a database using `flock` for its own safety must be told +whether that lock spans mounts. It does not, and saying so plainly is what +keeps someone from running two Postgres instances against one volume. + +## Acceptance criteria + +1. Two read-write mounts of one volume: the second fails, naming the first. +2. `kill -9` the writer; after 60s another host mounts with `--steal`; all data + written before the crash is intact. +3. `--steal` against a live, renewing lease is refused. +4. A fenced writer goes read-only within one renewal interval, loses no + journalled data, and `export-divergent` recovers every post-divergence file. +5. Ten concurrent read-only mounts during a heavy write load: none observes a + torn snapshot; staleness stays within `root_poll_ms` + anchor latency. +6. Reader cache invalidation on a root advance touches O(changed paths), not + O(volume) — asserted via a counter. +7. Network partition of the writer for 5 minutes: it fences itself; the volume + stays available read-only; no split-brain root updates land. +8. `statfs` reports the single-writer model and current staleness. +9. `flock` within one mount excludes correctly; the docs state it does not + across mounts. + +## Risks + +- **Users expect multi-writer and will be disappointed.** Two people editing + files on one volume is an obvious ask. *Mitigation:* be explicit everywhere — + CLI, docs, `statfs`. A clear limit beats a vague promise. Note that S3-style + access (CIP-012) has no such limit for object writes, which covers a good + share of the demand. +- **CoinPay latency in the lease renewal path.** A slow anchor could fence a + healthy writer. *Mitigation:* gossip is the fast path; renewal starts at + one-third of the term, giving three attempts before expiry. +- **Clock skew.** Leases are wall-clock. A writer with a fast clock may believe + it is fenced early; a slow one may believe it still holds a lease it lost. + *Mitigation:* the epoch check is the real enforcement and is clock-free — + time only decides when a *new* holder may take over, and the 60s term absorbs + ordinary skew. Nodes with skew beyond 5s are flagged by `doctor`. +- **`--steal` used carelessly on a live-but-partitioned writer.** *Mitigation:* + refuse while unexpired; the epoch fence makes the outcome safe even when it + is used, at the cost of the divergent-export dance. + +## Future work: real multi-writer + +If demand justifies it, the path is sharded roots: split the volume's namespace +into subtrees, each with its own root pointer and lease, so writers working in +different directories never contend. Cross-subtree `rename` then needs a +two-phase protocol. That is a substantial project — CRDT-style directory merge +plus a distributed rename protocol — and should be its own DIP, not an +extension of this CIP. + +## Open questions + +- Is 60 seconds the right term? Shorter recovers faster from crashes; longer + tolerates worse networks. 60s with 20s renewal is a starting point, not a + measured one. +- Should read-only mounts optionally pin a snapshot (never auto-advance) for + reproducible job inputs? Cheap to add and useful for training runs. +- Should `--steal` require a second confirmation when the holder was seen alive + within the last term? diff --git a/docs/prds/011-encryption-keys.md b/docs/prds/011-encryption-keys.md new file mode 100644 index 0000000..da5690b --- /dev/null +++ b/docs/prds/011-encryption-keys.md @@ -0,0 +1,204 @@ +--- +cip: 011 +title: "Client-side encryption and key management" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 (`private` tier), DIP-0018 (crypto stack precedent) +depends-on: 007 +blocks: +implementation: +estimate: "2–3 weeks" +--- + +## Summary + +Encrypt file content and metadata on the client so storage providers hold only +ciphertext. DIP-0012 names sovereignty as one of the five reasons to choose +c0mpute over R2, and it is the only one of the five that a hyperscaler +structurally cannot match — but it is worthless as a claim until the bytes +leaving the client are actually opaque. + +## Motivation + +Today every shard is plaintext on a stranger's disk. That is acceptable for +public data and unacceptable for anything else, and it makes "sovereignty" a +marketing word rather than a property. + +The crypto stack is already chosen and vendored. DIP-0018's secure-chat plugin +brought in `aes-gcm`, `x25519-dalek`, `ed25519-dalek`, `hkdf`, `argon2`, +`zeroize` — all at workspace level in `Cargo.toml`. This CIP applies existing, +already-reviewed dependencies rather than introducing new ones, which is most of +why it is a 2–3 week phase and not a 6-week one. + +## Goals + +- File content encrypted client-side before erasure coding. +- File *names* and directory structure encrypted — metadata leaks plenty. +- Per-volume keys derived from the CoinPay DID or a passphrase. +- Key rotation without re-uploading all data. +- Sharing a volume, or a subtree, with another DID. +- Deduplication that still works within a volume. + +## Non-goals + +- Encrypted compute. A worker transcoding a video needs plaintext; DIP-0012 + already says `private` does not apply to workloads that process content. +- Post-quantum for storage at rest in v1. The dependencies are present via + DIP-0018; see Open questions. +- Hiding file *sizes* or access patterns. See Out of scope. + +## Design + +### Key hierarchy + +``` +DID master key (CoinPay, or passphrase via argon2id) + │ + ├─ HKDF("c0mpute/storage/volume/" || volume_id) ──► Volume Key (VK) + │ │ + │ ├─ HKDF(VK, "content" || generation) ──► Content Key (CK_g) + │ ├─ HKDF(VK, "names") ──► Name Key (NK) + │ └─ HKDF(VK, "convergent") ──► Convergence Secret (CS) + │ + └─ per-recipient X25519 wrap ──► shared volume access +``` + +Only the master key is ever stored (in the OS keyring, or derived from a +passphrase). Everything else derives, so there is no key database to lose. + +### Content encryption + +Each chunk is encrypted **before** erasure coding, so providers hold ciphertext +shards and the RS math is unchanged: + +``` +nonce = blake3(chunk_plaintext_hash || generation)[0..12] +ciphertext = AES-256-GCM(CK_generation, nonce, chunk_plaintext) +shard_* = RS_encode(ciphertext) +``` + +The manifest records the *ciphertext* hash — which is what the network needs +for integrity — and the inode records the plaintext hash for the client's own +verification. Both checks stay intact end to end. + +### Convergent encryption, and its cost + +Encrypting identical chunks under a random nonce destroys dedup. Deriving the +nonce from the plaintext hash (above) preserves it — identical plaintext yields +identical ciphertext, so dedup works **within a volume**. + +The known weakness of convergent encryption is the confirmation-of-file attack: +someone who guesses a chunk's plaintext can confirm you store it. Mixing the +per-volume `CS` into the derivation limits the attack to holders of the volume +key, at the cost of losing cross-volume dedup. + +That trade is right for a `private` tier — cross-volume dedup is a provider-side +saving, and confidentiality is what the tier is sold on. `standard` and `hot` +keep global dedup by not encrypting. So the tier choice and the dedup behaviour +are the same choice, and the docs should say so rather than surprising anyone +with a storage bill. + +### Metadata encryption + +Filenames leak a great deal. Directory entries are encrypted with `NK` using +AES-SIV (deterministic, so lookup by name works without decrypting a whole +directory): + +``` +stored_name = base64url(AES-SIV(NK, parent_ino || plaintext_name)) +``` + +Determinism per parent means the same name in different directories encrypts +differently, so structure is not inferable across directories. Inode bodies — +sizes, times, modes — are encrypted with `CK` as ordinary content. + +What remains visible to a provider: the shape of the HAMT (roughly, how many +entries exist), object and chunk sizes, and access timing. That is a real +residual leak and should be stated plainly in the docs rather than glossed. + +### Key rotation + +Rotation bumps `generation` and derives a new `CK`. Existing chunks stay under +their old generation — recorded per-extent — so **rotation is O(1), not a +re-upload of the volume.** New writes use the new key. An explicit +`c0mpute storage rekey --rewrite` re-encrypts everything for the case where the +old key is believed compromised, and that one does cost a full rewrite. + +### Sharing + +Grant another DID access by wrapping the VK to their X25519 public key and +storing the wrapped blob in the volume's access list: + +``` +c0mpute storage share vol_7f3a --with did:coinpay:... --mode ro +``` + +Subtree sharing wraps a key derived at that subtree instead. Revocation +requires rotation with `--rewrite` to be meaningful — anyone who held the old +key may have kept the plaintext. The CLI must say this at revoke time; a +revocation that silently does not revoke is worse than none. + +### Performance + +AES-256-GCM with AES-NI runs at several GB/s per core, well above the network +path, so encryption is not the bottleneck. It does add CPU on a node that may +be running inference — same concern as FastCDC in CIP-007, same mitigation +(cgroup the mount). + +## Acceptance criteria + +1. With `--encrypt`, no shard on any provider contains recognisable plaintext + (grep a known string across every node's chunk store: zero hits). +2. Directory listings on a provider reveal no plaintext filenames. +3. Writing the same 100 MB file twice in one volume stores it once (convergent + dedup); in two different volumes, twice. +4. `rekey` without `--rewrite` completes in under a second on a 1 TB volume and + new writes use the new generation while old data still reads. +5. `rekey --rewrite` re-encrypts everything and old-key readers fail. +6. A shared DID can read; after revocation plus rewrite, it cannot. +7. Losing local state and recovering from the DID key alone restores full + plaintext access. +8. Encrypted throughput is within 10% of unencrypted on the same hardware. +9. Keys are zeroized on drop (`zeroize` derive present on every key type). + +## Risks + +- **Lost key means lost data, permanently.** No recovery, by design. + *Mitigation:* mandatory acknowledgement at volume creation; optional + Shamir-split escrow to N recipient DIDs; loud, repeated documentation. +- **Convergent encryption's confirmation attack.** *Mitigation:* per-volume + convergence secret; document the residual risk honestly rather than claiming + it is eliminated. +- **Encrypted metadata makes server-side features impossible.** No server-side + search, no provider-side dedup across customers, no listing without the key. + *Mitigation:* accepted — it is the point of the tier. +- **Rolling our own construction.** *Mitigation:* use standard primitives in + standard modes only; no novel crypto; commission a review before any + `private`-tier data is accepted from a paying customer. +- **Deterministic name encryption leaks equality of names within a directory.** + Accepted; AES-SIV is chosen precisely for lookup, and the alternative is + decrypting whole directories per lookup. + +## Estimate + +**2–3 weeks.** ~0.5 week key hierarchy and derivation, 0.5 week content +encryption in the chunk pipeline, 1 week metadata and name encryption with +lookup, 0.5 week sharing and rotation, 0.5 week tests and adversarial review. + +## Out of scope + +- Hiding file sizes (would need padding, and it is expensive). +- Hiding access patterns (would need an ORAM-shaped design). +- Hiding volume existence or total size from providers. + +## Open questions + +- Post-quantum wrapping for shared keys? DIP-0018 already brought a hybrid + classical+PQ stack in for secure-chat; matching it here is mostly plumbing and + matters for data with a long confidentiality horizon. +- Should `private` be the default tier rather than opt-in? It costs dedup and + makes support harder, but "encrypted by default" is a much better promise. +- Where does the master key live on a headless worker with no OS keyring? diff --git a/docs/prds/012-s3-gateway.md b/docs/prds/012-s3-gateway.md new file mode 100644 index 0000000..905a943 --- /dev/null +++ b/docs/prds/012-s3-gateway.md @@ -0,0 +1,200 @@ +--- +cip: 012 +title: "S3-compatible gateway" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012, DIP-0013 (BYOS positioning) +depends-on: 004 +blocks: +implementation: +estimate: "2–3 weeks" +--- + +## Summary + +Speak enough of the S3 API that existing tools — `aws s3`, `rclone`, `s3fs`, +every SDK, every backup product — work against a c0mpute volume with only an +endpoint change. This is the cheapest distribution the storage product can buy. + +## Motivation + +`docs/storage-pricing.csv` lists `api_style` for every competitor. Almost all of +them say **S3**, including Storj, Wasabi, B2, Hetzner, and Filebase. S3 is the +lingua franca of object storage, and a network storage product without it asks +every prospective customer to write an integration before they can evaluate it. + +It is also strictly less risky than the mount: no POSIX semantics, no +concurrent-writer problem (S3 has never promised more than last-writer-wins), +no kernel involvement. It depends only on CIP-004, so it can be built **in +parallel with the entire filesystem track** and shipped first if the mount +slips. + +DIP-0013 says BYOS3 is the default and c0mpute storage is opt-in. Speaking S3 +makes "opt in" a one-line config change rather than a project. + +## Goals + +- The S3 operations real tools actually use. +- SigV4 authentication, mapped to CoinPay DIDs. +- Multipart upload for large objects. +- Presigned URLs for browser upload/download. +- Interoperability with the same volumes the mount uses. + +## Non-goals + +- Complete S3 API coverage. Versioning, lifecycle rules, replication, + object-lock, inventory, analytics, website hosting, and event notifications + are all out. +- IAM policy semantics. Access is per-volume by DID, not policy documents. +- Byte-compatible error XML for every edge case; correct codes for the common + ones. + +## Design + +### Mapping + +| S3 concept | c0mpute | +|---|---| +| Bucket | Volume (CIP-004) | +| Key | Path within the volume | +| Object | File inode + extents (CIP-007) | +| ETag | blake3 hash, hex (**not** MD5 — see below) | +| Storage class | Tier: `STANDARD`→`standard`, `REDUCED_REDUNDANCY`→`hot`, `GLACIER`→rejected | + +Buckets and volumes being the same thing is what makes the gateway and the +mount interoperable: write via `aws s3 cp`, read through the FUSE mount, and +vice versa. That is a genuinely useful property and worth protecting in tests. + +### Operations + +``` +Service ListBuckets +Bucket CreateBucket, DeleteBucket, HeadBucket, ListObjectsV2, ListObjects +Object GetObject (+ Range), PutObject, HeadObject, DeleteObject, + DeleteObjects, CopyObject +Multipart CreateMultipartUpload, UploadPart, CompleteMultipartUpload, + AbortMultipartUpload, ListParts, ListMultipartUploads +Presign GET and PUT +``` + +That set covers `aws s3 sync`, `rclone`, `s3fs`, restic, and the major SDKs. + +### The ETag problem + +S3 clients expect an ETag that is the MD5 of the object (for single-part +uploads), and some verify it. We hash with blake3 everywhere and have no reason +to compute MD5 over every byte we store. + +- Return the blake3 hex as the ETag, which is opaque to well-behaved clients. +- Compute MD5 lazily **only** when a client sends `Content-MD5` or + `x-amz-content-sha256` and expects verification, and cache it in the inode's + xattrs. +- Multipart ETags already use S3's `-` form, which no client + can interpret as a plain MD5, so mimic that shape for multipart. + +Document it. `rclone` and the AWS CLI are fine with opaque ETags; a minority of +tools that recompute MD5 client-side will complain, and that is a known, +bounded incompatibility rather than a surprise. + +### Authentication + +SigV4 with the access key ID being a DID-derived identifier and the secret +being a per-volume derived API secret: + +``` +c0mpute storage credentials create vol_7f3a --mode rw + access_key_id: C0MP7F3A9C2EEXAMPLE + secret_access_key: ... + endpoint: https:///s3 +``` + +SigV4 verification is standard; the credential lookup resolves to a DID and a +volume, and every request is authorised against that pair. Presigned URLs use +the same secret with S3's standard query-parameter scheme, so browser upload +flows work unmodified. + +### Multipart upload + +Maps cleanly onto the extent model (CIP-007): each part is chunked and uploaded +independently, then `CompleteMultipartUpload` assembles the extent tree in part +order. Parts are staged as ordinary content-addressed chunks, so an aborted +upload leaves only unreferenced chunks for CIP-004's GC. + +Minimum part size 5 MiB, matching S3, except the final part. + +### Consistency + +S3 has promised read-after-write consistency since 2020, and CIP-004's atomic +root advance provides it naturally — an object is invisible until the root +advances, then fully visible. + +`ListObjectsV2` reads a single snapshot, so a listing is a consistent +point-in-time view. That is *stronger* than S3 and costs nothing given the +design. + +Concurrent `PutObject` to the same key is last-writer-wins, as in S3. Note this +does **not** require CIP-010's write lease: object writes touch disjoint keys +and the root advance is a compare-and-set, so a losing writer simply retries. +Only the POSIX mount needs exclusive leases. This is why the S3 track can ship +independently, and it is worth stating clearly because it looks like a +contradiction otherwise. + +### Deployment + +Runs in the existing axum gateway, on a `/s3` prefix or a dedicated port, so it +inherits the current TLS and operational setup. An operator can also run a +public S3 endpoint for their own volumes. + +## Acceptance criteria + +1. `aws s3 sync ./dir s3://vol_7f3a/prefix/` then `aws s3 sync` back produces + byte-identical files. +2. `rclone check` between a local dir and the bucket reports zero differences. +3. `restic` initialises a repository, backs up, and `restic check` passes. +4. `s3fs` mounts the bucket and passes basic read/write. +5. A 5 GB multipart upload completes and `HeadObject` reports the right size; + an aborted multipart leaves no referenced data. +6. Presigned GET and PUT work from a browser with correct CORS. +7. An object written via S3 is visible in the FUSE mount at the same path with + correct size and mtime, and the reverse. +8. `ListObjectsV2` with 10k keys paginates correctly and each page derives from + one snapshot. +9. SigV4 with a bad signature returns `SignatureDoesNotMatch`; expired + presigned URLs return `AccessDenied`. +10. Concurrent PUTs to one key from 10 clients: one wins, no corruption, all + receive a well-formed response. + +## Risks + +- **S3 compatibility is a long tail.** Tools depend on undocumented behaviours; + "S3-compatible" invites bug reports forever. *Mitigation:* publish the + supported operation list explicitly, test against the four named tools, and + treat anything outside the list as unsupported rather than broken. +- **ETag/MD5 mismatch breaks a minority of clients.** *Mitigation:* lazy MD5 on + demand, documented. +- **Latency.** S3 clients expect sub-100 ms; CIP-001 budgets 200–500 ms for + p2p reads. Some tools' default timeouts and retry behaviour will suffer. + *Mitigation:* aggressive metadata caching; document recommended timeouts; + gateway-side read-ahead for sequential `GetObject`. +- **Anonymous public buckets invite abuse.** *Mitigation:* no public-read ACLs + in v1; presigned URLs only. DIP-0016's abuse policy applies to anything + publicly served. + +## Estimate + +**2–3 weeks.** ~0.5 week SigV4 and credentials, 1 week core object operations +and listing, 0.5 week multipart, 0.5 week presigned URLs and CORS, 0.5 week +interop testing against the named tools. + +## Open questions + +- Should the gateway be a separate binary/service, or a role of the existing + daemon? A role is simpler; a separate service scales independently and is + easier to put behind a CDN. +- Bucket naming: expose raw volume ids (ugly, stable) or user-chosen names + (nice, needs a namespace and collision handling)? +- Is `GLACIER` worth mapping to a future cold tier, or is rejecting it cleaner + given CIP-001 rules cold storage out? diff --git a/docs/prds/013-databases-on-storage.md b/docs/prds/013-databases-on-storage.md new file mode 100644 index 0000000..25f30af --- /dev/null +++ b/docs/prds/013-databases-on-storage.md @@ -0,0 +1,228 @@ +--- +cip: 013 +title: "Databases on c0mpute storage: what we support" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-08-29 +updated: 2026-08-29 +implements: DIP-0012 +depends-on: 010 +blocks: +implementation: +estimate: "2–3 weeks (validation, tooling, docs — little new storage code)" +--- + +## Summary + +Determine, by measurement rather than assertion, which database workloads run +correctly on a c0mpute mount, and ship the tooling and documentation for the +ones that do. The deliverable is a supported-configuration matrix backed by +crash tests, plus first-class support for the two patterns that work well: +**database backup/WAL archiving** and **read-replica dataset distribution**. + +## Motivation + +"Storage for file sharing and dbs" is the ask, and the mount is read/write, so +people will absolutely put databases on it. The question is not whether to +allow that — it is whether we find out what breaks in a test harness or in a +customer's production data. + +Two facts set the boundary, and they come from earlier CIPs rather than from +caution: + +1. **`fsync` costs a network round trip.** CIP-008's table puts `network` mode + at 200–800 ms. A database that fsyncs per commit is limited to a few + transactions per second. That is a performance fact, not a correctness one. +2. **Cross-mount locking does not exist.** CIP-010 makes `flock` mount-local + and enforces one writer per volume. A database that relies on file locks to + arbitrate between processes on *different* machines has no protection at + all. + +The second is the dangerous one. SQLite's reputation for corruption on network +filesystems comes almost entirely from broken locking on NFS, not from slow +I/O. Our single-writer lease actually addresses the usual cause — but only if +the deployment respects it, which is exactly what documentation and tooling +have to enforce. + +## Goals + +- A tested support matrix: engine, configuration, verdict, measured numbers. +- Crash-consistency validation for anything marked supported. +- First-class WAL-archive and backup tooling. +- Clear, loud guidance against the configurations that corrupt data. + +## Non-goals + +- Making a network filesystem competitive with local NVMe for OLTP. It is not, + and no amount of work here changes that. +- A block-device interface (NBD). See Future work. +- Distributed/multi-writer database support of any kind. + +## Design + +### Support matrix (hypotheses to be confirmed by the test plan) + +| Engine / config | Verdict | Reasoning | +|---|---|---| +| **SQLite, WAL mode, single mount, `fsync=network`** | Expected **supported** | One writer enforced by lease; WAL is append-heavy, which suits FastCDC; commit latency ~1 round trip | +| SQLite, rollback journal | Discouraged | fsyncs more per commit; no correctness issue, poor performance | +| SQLite, multiple mounts | **Unsupported — corrupts** | Cross-mount locking does not exist. Blocked by the lease, but must be documented | +| SQLite, `PRAGMA synchronous=OFF` | **Unsupported** | Loses durability the storage layer is providing; corruption on crash | +| **DuckDB, read-only over a dataset** | Expected **supported** | Read-mostly analytics is the ideal fit: large sequential reads, no write path | +| DuckDB, read-write | Needs measurement | Large temp/spill files; may be fine with a local spill dir | +| **Postgres, `PGDATA` on the mount** | **Unsupported** | Assumes local-disk fsync/`O_DIRECT` semantics and per-file durability guarantees a network FS cannot honour; commit latency makes it unusable regardless | +| **Postgres, base backup + WAL archive to the mount** | **Supported — recommended** | Sequential, append-only, no fsync-per-commit. The right pattern | +| MySQL/MariaDB `datadir` on the mount | **Unsupported** | Same reasons as Postgres | +| LMDB / RocksDB on the mount | Unsupported | `mmap` shared-writable (LMDB) is out of scope in CIP-007; RocksDB's fsync pattern is hostile | +| **Turso / libSQL replica sync to the mount** | Expected supported | Already an embedded-replica model; the mount just holds the file | + +The two rows in bold that say "recommended" are the actual product here. +Everything else is either a measured yes or an honest no. + +### Pattern 1: WAL archive and backup (recommended) + +This is what most people asking for "databases on distributed storage" actually +need, and it plays to every strength of the design: append-only writes, +sequential reads, and content-addressed dedup across daily backups. + +```bash +# postgresql.conf +archive_mode = on +archive_command = 'test ! -f /mnt/c0mpute/wal/%f && cp %p /mnt/c0mpute/wal/%f' +``` + +Ship `c0mpute storage db-backup` wrapping the common cases: + +``` +c0mpute storage db-backup postgres --volume vol_7f3a --dsn ... --schedule daily +c0mpute storage db-backup sqlite --volume vol_7f3a --file app.db +c0mpute storage db-restore --volume vol_7f3a --to ./restored --at 2026-08-29T12:00Z +``` + +Point-in-time restore comes almost free from CIP-004's retained roots and +CIP-007's structural sharing: a daily backup of a 100 GB database that changes +1% stores ~1 GB of new chunks per day, not 100 GB. Dedup across backups is the +single most compelling storage economic in the whole product, and it should be +measured and published. + +### Pattern 2: read-replica dataset distribution (recommended) + +Write a database file once, mount it read-only on many workers, query in +parallel. CIP-010 gives unlimited read-only mounts with consistent +point-in-time snapshots, which is exactly the semantics a shared analytical +dataset wants. + +This is the compute-locality argument made concrete: a DuckDB or SQLite dataset +read by 50 c0mpute workers pays $0 internal egress (CIP-001) where R2 or B2 +would charge for every worker's read. + +### Pattern 3: single-writer OLTP (supported, with numbers attached) + +SQLite in WAL mode on a `fsync=network` mount, one writer. Correct, because the +lease enforces the single-writer assumption SQLite already requires. Slow, in a +way that must be quantified rather than hand-waved — the documentation should +carry the measured tps, not an adjective. + +Provide a tuned mount profile: + +``` +c0mpute storage mount vol_7f3a /mnt/db -o fsync=network,profile=sqlite +``` + +which sets a smaller chunk target (WAL frames are small), disables FastCDC on +`-wal` files (their boundaries are already frame-aligned), and pins the journal +to the fastest local device. + +### Test plan + +The matrix above is a set of hypotheses, and this is the work that turns it +into a supported-configuration list: + +1. **Crash consistency.** For each candidate config: run a write workload, cut + power (`dm-flakey`, `kill -9`, and network partition), then run the engine's + own integrity check (`PRAGMA integrity_check`, `pg_checksums`, `amcheck`). + 1000 iterations, zero corruptions required to earn "supported". +2. **Performance.** `pgbench`, `sqlite-bench`, and a DuckDB TPC-H subset across + all three fsync modes, published as a table with real numbers. +3. **Lease enforcement.** Attempt the corrupting configurations deliberately — + two mounts, one database — and confirm CIP-010 blocks them. +4. **Backup/restore.** Restore correctness at multiple points in time; measure + dedup ratio across 30 daily backups. +5. **Long-running soak.** Two weeks of continuous write load with induced node + churn and repair, confirming no corruption under CIP-005 activity. + +Anything that fails crash consistency is documented as unsupported and, where +possible, **actively refused** — `profile=sqlite` should reject +`synchronous=OFF` rather than allowing a foot-gun. + +### Guardrails + +Documentation alone will not stop someone pointing `PGDATA` at the mount. + +- The mount detects known database file signatures (`PGDATA/PG_VERSION`, + `ibdata1`, LMDB `data.mdb`) appearing in an unsupported configuration and + emits a loud warning to `status` and the system log. +- `--allow-unsupported-db` exists to silence it, because someone will have a + good reason and refusing outright is paternalistic. But it must be a + deliberate act, not a default. + +## Acceptance criteria + +1. The support matrix is published with measured numbers in every row, and no + row says "should work" without a test behind it. +2. Every configuration marked supported survives 1000 crash iterations with + zero integrity-check failures. +3. `db-backup postgres` + `db-restore --at