perf(storage): compact PieceV2 storage

[Kubuxu]

Summary

Implementation of the compact two-slot PieceV2 storage prototype, related to #286.

New compact-state datasets store pieces in a contiguous per-dataset array:

struct PieceV2 {
    bytes32 root;
    uint256 metadata;
}

metadata packs padding, tree height, leaf count, and the Fenwick partial sum into one storage slot.

Changes

• Hardened canonical PieceCIDv2 validation and decoded (padding, height, root) in one pass.
• Added compact PieceV2 storage while retaining the existing declarations for physical layout safety.
• Moved piece addition, proof verification, lookup, getters, pagination, CID search, scheduling, removal, and cleanup to compact storage.
• Preserved piece IDs, public ABI behavior, events, and listener callback arguments for compact-state datasets.
• Added coverage for packing bounds, Fenwick semantics, proofs, pagination, removal, cleanup, and raw storage reclamation.
• Measured steady-state addPieces storage activity for 1, 4, 16, and 32-piece batches.

Storage measurements

Each scenario creates a fresh dataset, adds one seed piece before recording, then records the measured addPieces call.

Batch	EVM-slot reads	EVM-slot writes	KAMT touched	KAMT modified	Newly occupied slots
1	10 → 12	7 → 4	11 → 7	6 → 3	5 → 2
4	21 → 25	28 → 13	24 → 7	19 → 3	20 → 8
16	69 → 85	112 → 49	72 → 8	67 → 4	80 → 32
32	133 → 165	224 → 97	137 → 9	132 → 5	160 → 64

Values are legacy → compact.

Findings

• Storage footprint: consistently 60% lower—5N → 2N newly occupied slots.
• EVM-slot writes: reduced from 7N to 3N + 1; reduction grows from 42.9% at one piece to 56.7% at 32 pieces.
• KAMT locality: compact contiguous records substantially reduce object activity as batches grow:
  • 4 pieces: 24 → 7 touched, 19 → 3 modified.
  • 16 pieces: 72 → 8 touched, 67 → 4 modified.
  • 32 pieces: 137 → 9 touched, 132 → 5 modified.
• EVM-slot reads: increase by 19.0–24.1%. These are repeated reads of already-accessed values, which are cached; they do not correspond to a comparable increase in KAMT object activity.
• Scaling: the compact layout modifies two shared objects plus roughly one compact-data object per 16 pieces, subject to alignment. The legacy representation modifies approximately four piece-data objects per piece.

At 32 pieces, the compact layout occupies 64 slots instead of 160, touches 9 KAMT objects instead of 137, and modifies 5 instead of 132.

[Kubuxu]

Benchmark results:

Optimised gas reduction versus baseline:

Ballast slots	Batch 1	2	4	8	16	32
0	9.9%	9.1%	27.5%	33.1%	36.2%	38.1%
1M	40.4%	57.4%	72.8%	82.3%	87.0%	89.5%
5M	40.3%	60.7%	73.6%	83.5%	88.3%	91.1%

Flat gas used

Ballast slots	Version	Batch 1	2	4	8	16	32
0	Baseline	17,217,928	22,999,322	42,969,779	64,817,453	112,656,392	205,053,901
0	Optimized	15,504,996	20,909,427	31,155,721	43,346,504	71,885,266	126,894,938
1M	Baseline	135,895,549	200,896,826	341,493,403	592,579,677	989,314,920	1,817,386,096
1M	Optimized	80,992,902	85,593,374	92,775,205	105,070,523	128,651,224	190,157,808
5M	Baseline	157,548,595	249,842,564	389,163,815	711,535,675	1,251,005,462	2,259,544,725
5M	Optimized	94,115,959	98,222,892	102,830,002	117,059,589	146,507,177	200,297,037

The optimisation largely removes ballast sensitivity: at 5M slots, batch 32 falls from 2.260B to 200.3M gas (91.1%, or 11.3× lower). Batch 64 is excluded because both versions revert at the Lotus event-size limit.

Benchmarks were performed in foc-devnet, ballasting down the PDPVerifier tree with the added dev-only balast(start, n) method creating sputtering of occupied slots.

[Kubuxu]

I plan to remove this doc. It was generated as part of LLM task guidance.

[Kubuxu]

@rvagg @wjmelements I would appreciate an early review. I don't expect the logic to change much, but it will get way messier when I make this backwards compatible.

[rvagg]

Batch 64 is excluded because both versions revert at the Lotus event-size limit.

But I think we can fix that here can't we because (a) we're emitting those events with uncapped arrays of piece CIDs and (b) the piece CID we're emitting are the larger legacy format, not the compact 64-byte format; so we could go quite high, at least for the PDPVerifier component (FWSS will impose its own limits), right?

[wjmelements]

Solidity does a lot of unnecessary checks unless you tell it not to. For example, did you know offset++ will do a uint256 overflow check?

[wjmelements]

This offset is added to the data's offset, but you can instead iterate using a raw pointer. Then instead of add(add(, it would just be root := mload(offset). I have an example of this with calldata in fvm-solidity (_cdReadArrayHeader) and also with memory (_writeCborArrayHeader).

For example, you would replace height = uint8(cid.data[offset++]) with

height := byte(0, mload(offset))
offset := add(1, offset)

[Kubuxu]

Thanks for the suggestion. I'm reluctant to implement more of asm optimisations here, as correctness matters and readability suffers significantly.

[wjmelements]

Correctness is checked by testing, and you should have good test coverage before doing any optimization.

Coming from C, I think one pointer parameter is more readable than two.

[wjmelements]

Is this ULEB128?

[wjmelements]

I have a prior assembly implementation of a ULEB128 read in _getOwnerActorId in fvm-solidity, (though it can assume the encoded number is a uint64)

[Kubuxu]

Yeah, it is ULEB128

[wjmelements]

Let's call it that instead of Uvarint so the endianness will be clear in the method name

[Kubuxu]

varint is term used everywhere within the multihash ecosystem
LEB128 is less specific: https://github.com/multiformats/unsigned-varint
and ULEB128 is not really an established thing

[wjmelements]

be sure to revert

[wjmelements]

I like this a lot. It's a huge improvement. You can perhaps provide a way to permissionlessly migrate, and once we are sure everything is migrated (which can be checked with a script), we can drop the old path.

[Kubuxu]

You can perhaps provide a way to permissionlessly migrate

I was thinking about leaving old datasets as they are and supporting only new datasets. Otherwise, we would have to keep track of datasets which have been migrated, partial migration states, so on.

[wjmelements]

You can perhaps provide a way to permissionlessly migrate

I was thinking about leaving old datasets as they are and supporting only new datasets. Otherwise, we would have to keep track of datasets which have been migrated, partial migration states, so on.

Yeah that might be necessary, but I think it can be piecewise. The backwards compatibility is already going to need to be able to distinguish these states. Distinguishing piecewise might be less overhead than distinguishing by data set for individual pieces, but on the other hand there might be savings if you can assume an entire data set is one way or the other. I didn't notice such a situation in my brief review yesterday though.

[Kubuxu]

The code is now backwards compatible through a datasetID cutoff.

I did some additional backward-compatibility checks beyond what is in tests.

• Rehearsed the full upgrade and migration against a prepared snapshot of the mainnet PDPVerifier state at epoch 6,251,280 using fvm-anvil.
• Compared legacy dataset and piece queries before and after the upgrade; all results matched.
• Verified that datasets created before migration continue using legacy storage and can still accept new pieces after the upgrade.
• Verified that datasets created at the cutover boundary use compact storage without touching the legacy mappings.
• Successfully deleted and cleaned up datasets using both storage formats.
• Checked roughly 5.5 million piece records from the early-repair database against the new CID validation logic. All passed, and the decoded raw sizes matched the stored values.

I didn’t find any backward-compatibility issues in these checks.

[Kubuxu]

I ran a Filecoin snapshot benchmark comparing the deployed legacy storage layout with the optimized compact layout using 1,024-piece datasets.

Removals are applied during nextProvingPeriod. The compact layout substantially reduces both that operation and subsequent proof costs.

Removal lifecycle

Scheduled removals	Operation	Legacy gas	Compact gas	Change
1	schedulePieceDeletions	77,567,624	81,705,016	+5.3%
1	nextProvingPeriod	463,004,407	240,960,202	-48.0%
1	provePossession — 5 challenges	185,953,204	101,444,392	-45.4%
32	schedulePieceDeletions	197,760,288	152,283,667	-23.0%
32	nextProvingPeriod	2,753,390,816	460,964,008	-83.3%
32	provePossession — 5 challenges	183,231,003	97,787,401	-46.6%
96	schedulePieceDeletions	382,838,618	252,093,235	-34.2%
96	nextProvingPeriod	7,142,448,164	811,948,603	-88.6%
96	provePossession — 5 challenges	184,633,835	100,597,572	-45.5%

The 96-piece case is the largest tested with adequate headroom: the legacy implementation already consumes 7.14 billion gas against Filecoin’s 10-billion block limit.

Other important measurements

Operation	Case	Legacy gas	Compact gas	Change
createDataSet	empty	123,076,468	123,280,376	+0.2%
addPieces	1 piece	174,824,288	104,081,347	-40.5%
addPieces	8 pieces	799,346,484	131,556,419	-83.5%
addPieces	32 pieces	2,548,376,069	216,751,448	-91.5%
getPieceCid	tail piece	19,602,345	24,729,807	+26.2%
getActivePiecesByCursor	32-piece page	226,066,144	132,714,635	-41.3%
findPieceIdsByCid	limit 32	213,416,170	58,433,616	-72.6%
getActivePiecesByCursor	after 129 removal holes	436,279,799	162,286,483	-62.8%
deleteDataSet	1,024 slots	94,626,110	77,991,600	-17.6%
cleanupPieces	1 piece	149,154,504	89,602,117	-39.9%
cleanupPieces	16 pieces	1,341,620,438	98,475,885	-92.7%
cleanupPieces	64 pieces	4,553,835,628	136,541,618	-97.0%
cleanupPieces	96 pieces	6,598,846,685	161,517,495	-97.6%
cleanupPieces	96 pieces and finalize	658,885,130	194,883,685	-70.4%

The only notable regressions are the single-piece removal scheduling case at +5.3% and getPieceCid at +26.2%. All larger write, traversal, proof, and cleanup operations show substantial savings.

[Kubuxu]

This is a bit ugly because it is where we clear out the piece's metadata, overloading sumTreeRemove to also mark the piece as removed.

[Kubuxu]

I also have an additional optimisation of nextProvingPeriod performing removals by batching the sumTree updates. It is 10-20% on top of the compact storage format, but it is quite ugly, so I didn't decide to go with it.