feat: add complete Flock Stage 3 compression for aggregate roots

[johnchandlerburnham]

Stack

This PR is stacked on jcb/sp1-compressor and should target that branch:

main
  └── jcb/aggregate-first
        └── jcb/sp1-compressor
              └── jcb/flock-stage3  ← this PR

The base branches supply the shard-to-root aggregation pipeline, canonical
Stage 2 aggregate root, and the existing SP1 terminal-compression experiment.
This PR adds an independent no-RISC-V Stage 3 backend: it verifies the complete
specialised Stage 2 relation directly in Flock over F128 with BLAKE3.

Summary

The resulting pipeline is:

closed ix_aggr root
  → canonical Stage 2 vk / claim / compact proof transport
  → typed AIR / logUp / PCS / FRI verifier witness
  → specialised Flock Fast128 relation
  → versioned, verified Stage 3 artifact
  → future Stage 4 Ethereum SNARK

Stage 3 is intentionally an off-chain intermediate proof. It removes the
general-purpose RISC-V verifier overhead and leaves Stage 4 with one fixed
Flock verification relation suitable for a universal-setup PLONK/FFLONK
development backend. A circuit-specific Groth16 endpoint remains available
once that relation stabilises and sub-kilobyte settlement is the overriding
constraint.

The complete production relation now proves and verifies successfully. The
latest prover work also removes padded witness generation from the critical
path: the release CLI-shaped round trip now takes approximately 2.12
seconds, versus 39.62 seconds for the previous release proving path,
without changing the 326,019-byte Stage 3 artifact.

Protocol configuration

The backend pins the proof-system configuration as part of the Stage 3
statement:

• Flock revision b310f35f35f68095537150a1c8c0a43caca9a29e;
• the F128 binary field;
• the Fast128 Ligerito profile;
• BLAKE3 Merkle commitments;
• a chained-BLAKE3 Fiat–Shamir transcript; and
• domain-separated production and conformance transcripts.

Changing the upstream revision or any of these parameters is a protocol
change and changes the configuration digest. Poseidon2 is not used anywhere
in this path.

Complete Stage 2 verifier relation

The Flock relation constrains all eleven registered verifier phases,
including:

• canonical verifier-key, claim, and proof encodings;
• typed witness shape, sparse circuit activation, and trace heights;
• the specialised Aiur AIR and verifier-key metadata;
• all 18 canonical Goldilocks claim words;
• lookup-message inversion and the complete logUp balance;
• exact chained-BLAKE3 transcript replay and field rejection sampling;
• Goldilocks and quadratic-extension arithmetic;
• AIR selectors, DAG evaluation, OOD composition, and quotient
  recombination;
• multi-matrix and multi-height PCS openings;
• BLAKE3 MMCS leaf hashing and authentication paths;
• every binary-FRI challenge, grinding draw, query index, fold, roll-in, and
  final-polynomial check; and
• the published Stage 2 root shared by the statement and proof constraints.

The BLAKE3 lowering supports multi-block rows and messages beyond one
1,024-byte chunk. Transcript sampling matches Plonky3 rejection sampling,
including the constrained chained refill.

Native Stage 2 verification remains an inexpensive fail-fast guard before
allocating the production relation. It is not part of the soundness boundary:
the Flock relation repeats the relevant verification constraints.

Statement and artifact binding

Stage3StatementV1 binds three independent values:

1. the canonical Stage 2 root digest;
2. the complete relation-manifest digest; and
3. the pinned Flock configuration digest.

Verification requires an externally expected Stage 3 statement. In
particular, the verifier does not accept a relation digest supplied only by
the prover. Deployments and the eventual Stage 4 verifier must pin the
expected relation digest.

Stage3ArtifactV1 adds strict versioned framing and bounded parsing. The
production payload carries the canonical Stage 2 transport, compiled-circuit
digest, and Flock proof bundle needed to reconstruct and verify the relation.

Production aggregate integration

The SP1 and Flock root commands share one canonical AggregateRootInputs
preparation path. This prevents the two compression backends from drifting
on:

• aggregate claim construction;
• recursion verifying-key selection;
• outer-claim serialization;
• compact-proof extraction; and
• FRI parameters.

A new optional flock FFI feature exposes the backend without adding Flock to
the normal Ix build.

Preflight a persisted aggregate root with:

IX_FLOCK=1 nix develop --command lake exe ix flock-root ROOT_ADDRESS \
  --mode preflight

Preflight performs all of the following without starting the cryptographic
Flock prover:

• validates that the root is a closed CheckEnv aggregate;
• natively verifies and expands the compact Stage 2 proof;
• constructs the complete typed AIR/PCS/FRI witness;
• builds and evaluates every Flock gate;
• constructs the relation manifest and Stage 3 statement; and
• reports transport sizes, advice geometry, table capacity, nu, gate-row
  counts, and all relevant digests.

After successful preflight, generate an artifact with:

IX_FLOCK=1 nix develop --command lake exe ix flock-root ROOT_ADDRESS \
  --mode prove --output root.stage3.flock

Proof mode preflights, proves the complete relation, checks that the resulting
statement exactly matches the preflighted root/relation/configuration,
cryptographically verifies the artifact, and only then installs it through an
atomic rename.

The Lake build traces the nested flock-stage3 Rust sources so changes to the
optional connector cannot silently reuse a stale static archive. The Nix
inputs also use public, immutable sources so CI does not depend on an
interactive GitHub credential.

Full FRI schedules

This removes the former eight-round implementation ceiling.

The accepted commit-phase length is derived from the initial FRI height and
remains bounded by MAX_LOG_HEIGHT = 31. With the current binary FRI
production parameters, schedules through the current 30-round maximum are
supported.

Circuit-level regressions construct and evaluate valid transcript-bound
relations at 9, 16, and 30 rounds. Negative tests reject a missing final round
and a corrupted deepest Merkle path.

Prover architecture and optimization

The initial complete prover was correct but spent almost all of its time
materialising padded Boolean witnesses. Every table has uniform 2^nu
capacity, while the current complete fixture uses only a small live prefix.
The allocating path constructed logical z, Az, and Bz vectors across the
entire capacity, packed them, copied them into the union, and later cleared the
large buffers.

This PR now uses a generic live-row in-place Boolean driver:

• rows are evaluated in parallel groups of eight, matching the lincheck
  stripe layout;
• z, Az, and Bz are written directly into Flock's union buffers;
• only declared rows and useful columns are evaluated;
• padding writes are skipped when the merged union declares padding unread;
• live rows still check (Az) * (Bz) = z before proving;
• all eleven production table slots use the in-place path; and
• an allocating-vs-in-place differential test checks observable buffer and
  stripe equality, including a partial final group.

Additional setup improvements include:

• compiled CSC lincheck circuits cached once per process and reused by proving
  and verification;
• Boolean table plans cached where their construction is nontrivial;
• freshly built R1CS matrices moved into TableType instead of deep-copied;
• a bounded one-entry exact-witness relation cache shared by preflight,
  proving, and post-hoc verification; and
• opt-in nested timing under IX_FLOCK_TIMING=1, with Flock's internal phase
  tracing available under PCS_TRACE=1.

The relation cache compares the complete witness for equality rather than
using a digest, so a hit cannot substitute a relation for different proof
values. It retains only the most recent relation and therefore cannot grow
without bound.

Measured complete proof

The ignored complete regression uses a real canonical multi-STARK proof with:

• an inactive leading circuit;
• active circuits at heights 8 and 4;
• an active preprocessed matrix;
• an 18-word claim lookup;
• nontrivial first-row and transition constraints; and
• two FRI queries.

The artifact remains byte-for-byte the same size after the prover rewrite:

Output	Size
Stage 3 artifact	326,019 bytes
Encoded production payload	325,893 bytes

The latest release run, shaped like the real CLI (preflight → prove → verify → negative checks), reported:

Phase	Time
Fixture setup	0.005 s
Relation preflight/setup	1.896 s
Proof generation after preflight	0.180 s
Artifact encode/decode	<0.001 s
Valid cryptographic verification	0.017 s
Corrupted-proof rejection	0.018 s
Complete round trip	2.116 s

Within proof generation, the Flock prover itself took 177.84 ms. The largest
improvements versus the preceding release trace were:

Internal phase	Before	After
Live/padded witness generation	566.46 ms	34.57 ms
Witness-buffer give-back	159.70 ms	1.47 ms
Complete Flock prove	1,140.48 ms	177.84 ms

Counting both one-time relation setup and proof generation, the previous
39.615-second release proving path is now approximately 2.076 seconds: about
a 19× end-to-end improvement. The distinction between setup and proof is
reported explicitly because the immutable relation can be reused, while a
one-shot process must still pay setup once.

This fixture is deliberately complete but small; it is not a substitute for
the pending full production ix_aggr capacity and memory run. Stage 3 is also
not intended for Ethereum calldata. Stage 4 must compress verification of this
fixed relation into the sub-kilobyte terminal proof.

Run the expensive regression explicitly with:

IX_FLOCK_TIMING=1 PCS_TRACE=1 \
  cargo test -p flock-stage3-host --release \
    real_stage2_production_artifact_round_trip -- --ignored --nocapture

Validation performed

The final optimization pass was validated with:

• cargo check -p flock-stage3-host;
• cargo test -p flock-stage3-host:
  • 54 passed;
  • 13 expensive tests ignored;
• the release complete-production regression above;
• successful valid proof verification and corrupted-proof rejection;
• the allocating-vs-in-place differential witness test;
• cargo fmt --all; and
• git diff --check.

The broader branch integration also includes:

• deep binary-FRI relation evaluation at 9, 16, and 30 rounds;
• real Stage 2 root lowering through the public preflight API;
• cargo clippy -p flock-stage3-host --all-targets -- -D warnings;
• cargo check -p flock-stage3-host --all-targets;
• cargo check -p ix-ffi;
• cargo check -p ix-ffi --features flock;
• cargo clippy -p ix-ffi --features flock -- -D warnings;
• nix develop --command lake build Ix.Cli.FlockRootCmd Main;
• IX_FLOCK=1 nix develop --command lake exe ix flock-root --help; and
• graceful invalid-address handling for both flock-root and
  compress-root.

Scope and remaining work

The relation is deliberately specialised to the current Ix protocol:

• 18 claim words;
• binary FRI;
• cap height zero;
• constant final-polynomial configuration; and
• an exact circuit activation and height shape.

Before freezing a deployment relation, we still need to:

• run preflight and proof generation on a full production ix_aggr root;
• record peak memory, proving time, relation capacity, and artifact size;
• measure capacity across a representative aggregate-proof corpus;
• add production-sized differential vectors with nonzero grinding;
• independently review the local Boolean R1CS tables and Flock soundness
  profile; and
• export canonical fixed-verifier inputs for Stage 4 witness generation.

The remaining local setup bottleneck is immutable shape construction: in the
latest run, table declaration and final circuit wiring consumed almost all of
the 1.896-second preflight. The exact-witness cache eliminates repeated work
for the CLI's preflight/prove/verify sequence. Reuse across different aggregate
proof values with the same structural layout will require separating
per-proof input/public extraction from Flock's immutable CircuitShape setup.

The next architectural boundary is Stage 4: compile verification of this fixed
Flock relation into the universal-setup FFLONK/PLONK development backend,
measure constraints and Ethereum gas, and retain the ability to switch the
same stable statement to a circuit-specific Groth16 endpoint if required.