Add ADR-0005 client-side earned reward eligibility gate#147
Closed
grumbach wants to merge 25 commits into
Closed
Conversation
On fast links the store adaptive-concurrency limiter sat pinned at its cold-start floor of ~8 for a whole upload, capping throughput. Three compounding causes, all fixed here: 1. Transport quorum-shortfall pollution. A close-group PUT that misses quorum only because dead/stale relayed peer addresses could not be dialled surfaced as `InsufficientPeers` -> `NetworkError`, counting against `success_rate` and cutting the cap. This is remote peer churn (same root cause as V2-551), not local backpressure. Split per-peer failures into timeout (genuine backpressure) vs dial/relay churn, and route the aggregate: any timeout keeps `InsufficientPeers` (still cuts the cap), a pure dial-churn shortfall surfaces the new neutral `Error::CloseGroupShortfall` (classified `ApplicationError`), and an app-only shortfall keeps its representative error (V2-468). The new variant stays a recoverable/deferred shortfall in the merkle retry. 2. Asymmetric increase/decrease gate. A Decrease fires every `min_window_ops` (8) samples but an Increase only every `window_ops` (32), and a Decrease zeroed the increase counter, so a few-percent shortfall permanently starved growth. Retain the increase credit across a Decrease AND relax the store `success_target` 0.95 -> 0.88 so normal per-chunk noise reads as a healthy window; genuine congestion (larger shortfall / timeout ceiling) still cuts the cap. Store-scoped; quote keeps the classic gate. 3. Per-wave/per-file cap snapshot. Store fan-out used `buffer_unordered(cap)` read once per wave/file, so mid-flight limiter growth never reached in-flight work. `merkle_store_with_retry` and the wave `store_paid_chunks_with_events` now roll a `FuturesUnordered` that re-reads the cap as each slot frees (the byte budget still bounds the wave path). Covers the merkle wave, whole-file, and deferred-retry paths. All logic is in ant-core with unit tests for each fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-out The whole-file merkle upload stored chunks in sequential fixed 64-chunk waves with a hard barrier between them. On a churny network the wave barrier is the throughput bottleneck: a handful of chunks whose close-group peers are dead/stale relayed addresses take minutes to dial/revalidate, and the whole wave cannot advance until its slowest chunk finishes — so the next wave's chunks don't even begin peer resolution. A production 2.36 GB / 609-chunk upload spent ~70% of its wall-clock in these inter-wave gaps (measured on STG-UL-01), while the V2-554 store limiter had ramped to 64 and the actual store bursts hit ~50 concurrent. Store concurrency was no longer the long pole; the wave serialization was. Remove the waves. The payment/proofs are already computed up front, so the store phase only needs a memory bound, not batching. Store the whole file as a single cap-bounded fan-out: - `merkle_store_with_retry` now takes addresses and a `store_one(addr)` that acquires its own body (reads it from the spill on demand), so only the <=cap in-flight stores hold a body. The store cap maxes at 64, so peak resident memory stays ~64 x MAX_CHUNK_SIZE — the same bound the 64-chunk waves gave — with no barrier. A slow straggler now overlaps with the rest of the file instead of stalling it. - `merkle_deferred_retry` drops its per-batch `read_bodies`/`batch_size` memory-bounding (the cap now bounds memory) and stores each round's whole pending set in one pass. - `upload_merkle_from_spill` (renamed from `upload_waves_merkle`) makes a single store call over all chunks instead of looping waves; deferred retry, stats, progress, and fatal/PartialUpload handling are preserved. New tests: the store pass has no barrier (a blocked straggler must not deadlock the others) and holds at most `cap` bodies in flight (memory bound). This overlaps but does not fix the underlying dead-relay resolution latency (V2-551); it stops that latency from serializing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…event Address PR WithAutonomi#137 review (Hermes / @dirvine): - The de-waved `upload_merkle_from_spill` fan-out sized concurrency straight from `store_limiter.current()`. The default store cap is 64 (~256 MB of spilled bodies, matching the old wave bound), but `AdaptiveConfig::sanitize` permits `adaptive.max.store` above 64, so a custom high max could hold many more full chunk bodies resident. Clamp the fan-out (and the deferred-retry rounds) via `merkle_store_cap`, capped at `MERKLE_STORE_MAX_IN_FLIGHT` (64), so peak resident store memory stays at the wave-era ~256 MB regardless of the configured max. Throughput is unaffected at the default cap. - The de-wave removed the per-wave `UploadEvent::WaveComplete` emission, but the event is a public phase-completion signal consumed by the CLI progress adapter and external API clients. Emit a single terminal `WaveComplete` ("wave 1 of 1") after the store pass and deferred retries so consumers keying on it still see the store phase complete. Per-chunk `ChunkStored` events drive the bar as before. New test `merkle_store_cap_clamps_to_memory_bound`. cargo fmt + clippy clean; 374 ant-core lib tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the PR WithAutonomi#137 review discussion. `WaveComplete` had no real consumer: the CLI handler only mirrored `ChunkStored` (`set_position(stored)`), and the ant-ui Tauri backend maps it to `None` (never forwarded to the frontend). Emitting a synthetic "wave 1 of 1" from the de-waved merkle path to preserve it was therefore pointless. Remove the event entirely instead: drop the `UploadEvent::WaveComplete` variant, both emissions (the single/wave-batch path and the merkle terminal one added in the previous commit), and the redundant CLI match arm. Per-chunk `ChunkStored` drives all upload progress bars, so behaviour is unchanged. `UploadEvent` is an internal in-process type (derives only Debug/Clone, not serialised), so this is not an API-schema change; ant-ui pins ant-core by tag and will drop its own dead match arm when it next bumps. cargo fmt + clippy clean; 374 ant-core lib tests pass. BREAKING CHANGE: The `UploadEvent::WaveComplete` variant is removed from ant-core's public `UploadEvent` enum. Any consumer that matches on it must drop that arm; upload progress is fully covered by `UploadEvent::ChunkStored`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…upload-aimd-concurrency-stuck-at-cold-start-floor-8-on-fast improve merkle upload throughput: ramp store concurrency + de-wave the store phase
Production nodes are running out of disk space and rejecting chunk uploads. When free space at a node's data directory falls to 500MB (the same reserve at which ant-node itself refuses to store), the daemon now automatically stops the smallest co-located node and deletes its data directory to reclaim space for the others. The smallest node is chosen to minimise the re-replication cascade on the network. A node that is the only one on its partition is never evicted; that surfaces as critical fleet health instead, so an operator can act manually. Evicted nodes get a persisted `evicted` status with an explanatory reason that survives daemon restarts, are never restarted, and can be dismissed from the list. An extensible fleet-health layer (green/warning/critical) surfaces disk pressure ahead of time and names the next eviction candidate; it is shown in `ant node status` and served at GET /api/v1/health. - Add NodeStatus::Evicted + persisted EvictionRecord on NodeConfig (serde-default, backwards compatible) - Add daemon/disk.rs: per-partition measurement + smallest-node candidate selection (shared by the monitor and the health layer so they never disagree) - Add daemon/health.rs: extensible FleetHealth model, disk check first, fixed 500MB/1024MB internal constants (deliberately not user-configurable) - Add the eviction monitor (stop -> delete data dir -> persist marker -> emit event), NodeEvicted and FleetHealthChanged events, and GET /api/v1/health - Add `ant node dismiss <id>` and a fleet-health summary line in `ant node status` Tested: cargo test -p ant-core is green, including the new disk/health/eviction/types unit tests and the daemon integration test. The only workspace failure was an unrelated OOM/SIGKILL on the gigabyte-scale e2e_huge_file upload test (client data path, untouched by this change). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An evicted node's data directory has been deleted, so trying to start it fails with a confusing "No such file or directory", and `start-all` reports it as a failure. Treat evicted nodes as non-actionable: `start-all`/`stop-all` skip them silently (they stay visible as `Evicted` until dismissed), and single start/stop return a clear message pointing at `ant node dismiss`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Guards backward compatibility of node_registry.json across the evm-network and eviction changes: a registry written by an older daemon (carrying the removed network_id/metrics_port fields and lacking evm_network/eviction) must still load, with the unknown fields ignored and the new ones defaulted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On Windows a just-killed node can keep its data files open for a moment after exit (its LMDB memory map and its own copied ant-node binary), and antivirus/indexers can grab transient handles, so remove_dir_all fails with "access denied / in use". The eviction then stopped the node without reclaiming any space, and the reason text falsely claimed it was reclaimed. Retry the delete with bounded exponential backoff (~4.5s) so the OS has time to release the handles; on Unix the first attempt still succeeds. If deletion ultimately fails, the node is still marked Evicted (its process is gone) but the reason text and reclaimed_bytes reflect that no space was reclaimed, and the failure is logged at error level. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the review on WithAutonomi#138. - Eviction candidate sizing (disk.rs): rank and report by actual reclaimable disk usage. dir_size now sums allocated blocks (blocks() * 512 on Unix) instead of logical file length, so LMDB's sparse data.mdb no longer mis-ranks the "smallest" node or misreports reclaimed bytes. It also uses symlink_metadata and skips symlinks so the walk cannot escape the node tree via a link. Adds regression tests for sparse sizing and symlink skipping. - Start/evict race (supervisor.rs): a concurrent POST /nodes/{id}/start could spawn a node during its stop/delete window. evict_node now makes the node unstartable *before* that window, via an in-memory `evicting` flag set under the supervisor lock (atomically excluding start_node) and a persisted eviction marker written first (which also survives a daemon crash mid-delete). Rollback is intentionally not attempted: once stopped, the node stays Evicted. Adds a regression test. - CI: apply rustfmt and fix two rustdoc intra-doc links (a private-item link and an unresolved EvictionRecord link). The security-audit failure (quinn-proto RUSTSEC-2026-0185 via saorsa-transport) and the is_running/UpgradeScheduled observation are pre-existing on main and out of scope here, per the review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-low-disk feat: automatic node eviction on low disk
…6.07.1 chore(release): release ant-client from main -> 0.2.10,0.3.0
Implement the client half of ADR-0003: the client pays nothing it cannot resolve. Before paying, it runs the full binding check on every quote and candidate, so a node that overstates its storage to inflate its price is dropped before any on-chain payment instead of after. For each single-node quote and each merkle candidate, the client now: - verifies the quote's own ML-DSA-65 signature and that it is for the requested content; - checks the (committed_key_count, commitment_pin) shape, the count cap, and that price == calculate_price(count) by exact recomputation; and - for a bound quote, fully resolves the shipped commitment: parse it, check it is bound to the quoting peer, verify its signature, that it hashes to the quote's pin, and that its key_count matches the claimed count. Anything unresolvable, withheld, off-curve, or contradictory is dropped before payment, exactly as the storer would reject it. The verified commitments are then forwarded as sidecars in the PUT bundle (single-node and merkle) so storers cross-check synchronously. Sidecar blobs are size-capped before parsing. Pricing and the commitment verification come from ant-protocol, so client and node never disagree. Adds an e2e test (real QUIC nodes + Anvil) proving bound quotes are shipped, priced, fully resolve, and round-trip; and the off-curve / forged-commitment rejection is covered in unit tests. Depends on the evmlib, ant-protocol, and ant-node ADR-0003 releases; will not build standalone until those publish.
[patch.crates-io] points evmlib/ant-protocol at their PR branches; ant-core points the ant-node dep (runtime + dev) at the matching ant-node ADR-0004 branch (published versions version-collide with the patch). At release: revert ant-node to a published version, drop the patch block, bump ant-protocol pin.
…ote flow; renumber ADR-0003 -> ADR-0004 StoreQuote carries the commitment sidecar end-to-end (resolve-before-pay); get_store_quotes family + median helpers + downstream consumers updated.
The renamed ADR-0004 e2e target was auto-discovered but never listed in the CI E2E command, so ant-client could go green without exercising the main ADR-0004 end-to-end coverage. Add --test e2e_adr0004.
Point the ant-node / ant-protocol / evmlib git-branch deps at the freshly clean-rebased ADR-0004 revisions (ant-node ff7a547 — the audit-code merge against current main resolved correctly; ant-protocol 3a4eb638 = 2.3.0; evmlib a7ac7e26 = 0.9.0). Recovers the workspace build now that ant-node's branch compiles again.
Add negative unit tests for the three resolve-before-pay commitment sub-checks that previously had no coverage (deleting any of them still passed all tests): invalid commitment signature, commitment that does not hash to the quote's pin, and quote count disagreeing with the committed key_count. Each constructs a validly-signed, peer-bound commitment and breaks exactly one field to isolate its check. Rename binding_rejects_garbage_and_wrong_pin_commitment to binding_rejects_unparseable_and_peer_unbound_commitment to reflect what it actually exercises (the 'wrong pin' case failed peer-binding first), and fix the stale 'adr-0003' e2e test payload strings.
Once nodes rotate their first storage commitment, every merkle candidate is bound and finalize_merkle_batch copied all 16 winner-pool commitment sidecars into EVERY per-chunk proof, growing it from ~128 KB to ~342 KB, past the storer's 256 KB payment-proof cap. Every merkle chunk PUT was rejected AFTER the on-chain payment, so forced-merkle uploads burned their payment and then failed every store attempt (DEV-01, 2026-07-06 19:38 UTC: both forced-merkle clients flipped to 100% failure the moment 855 nodes rotated their first commitment; single-node payments were unaffected). The sidecars are not needed in the bundle: the client fully resolves every candidate's shipped commitment before paying, and the storer's cross-check is best-effort with a gossip-cache / GetCommitmentByPin fallback. Drop the forwarding and the dead plumbing that threaded the sidecar map from pool collection into finalization. Receipts cached by pre-fix clients still carry the fat proofs, so a resume would replay the rejection: strip sidecars from cached merkle receipts on load, with an atomic (create_new tmp + fsync + rename) write-back that can never truncate the only copy of a paid receipt. New coverage: - e2e: forced-merkle upload against 35 committed nodes; failed with the exact production signature before this fix, passes after, against the unchanged 256 KB storer cap - unit: finalize_merkle_batch ships no sidecars (bound candidates) - unit: cached receipt strip + idempotency + non-merkle passthrough - unit: atomic overwrite lands and leaves no tmp sibling
The client decides who is payable. This adds the eligibility gate at quote collection (payee selection and payment verification untouched): a quoter is payable when more than half the other responders who know it vouch for a clean audited week at the quoted size. - eligibility: aggregate the quorum of signed audit reports into a two-tier decision. Tier 1 (size) needs a vouched clean week AT the quoted size; tier 2 (dues) relaxes size to keep the network paying under fast growth, and is safe because ADR-0004 independently forces a current-size proof at payment. Both tiers still exclude fresh identities (no dues) and caught cheaters (fenced/convicted). The eligibility map is keyed by (peer, quoted size) so one peer's small quote can never leak size-tier admission onto its larger quote. If too few are eligible, it prefers the ones with the most dues, then falls back to today's rules; the network never stalls. - merkle / quote: carry the client report_nonce on quote requests, collect the returned audit reports, and apply the gate at pool / close-group composition. - tests: unit coverage for the bar, both tiers, and the exclusions, plus an end-to-end suite (mixed pool under enforcement, fenced/unproven substitution, and fast-growth dues fallback). - Cargo: temporarily pin ant-protocol and ant-node to the ADR-0005 PR branches carrying the wire types and node tally; revert to published bumps once released. Based on the ADR-0004 client work in WithAutonomi#126.
Contributor
Author
|
Reopening with the correct stacked base (the ADR-0004 branch this depends on) so the diff shows the ADR-0005 changes only. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The client side of ADR-0005: earned reward eligibility, and the piece that actually decides who gets paid. The gate sits at quote collection, so payee selection and payment verification are untouched: a quoter is payable when more than half of the other responders who know it vouch for a clean audited week at the quoted size.
Changes
eligibility.rs(new) — aggregate the quorum of signed audit reports into a two-tier decision:merkle.rs/quote.rs— carry the clientreport_nonceon quote requests, collect the returned audit reports, and apply the gate at pool / close-group composition.Dependency pins (revert before merge)
Cargo.tomltemporarily pinsant-protocol(ant-protocol#19) andant-node(ant-node#169) to their ADR-0005 PR branches so CI can build the coordinated change. These revert to published version bumps once the wire types and node tally are released, same as the ADR-0004 cutover.Follow-up
Wiring
--test e2e_adr0005into.github/workflows/ci.ymlis left out of this PR (token lacks theworkflowscope); it should be added in a small follow-up so the new e2e suite runs in CI. It passes locally.Testing
cargo test -p ant-core --features test-utils --lib(426 pass, incl. 14 eligibility unit tests) and the ADR-0005 e2e suite (6 tests) pass locally against the ant-protocol#19 / ant-node#169 branches;cargo clippy --all-targets --features test-utils -D warningsclean;cargo fmt --checkclean.