docs(unixfs): match spec to fixtures and defaults - #553
Open
lidel wants to merge 1 commit into
Open
Conversation
The spec text drifted from what the linked fixtures contain and what boxo (Go) and helia (JS) already produce by default. No implementation changes; the text now describes the preexisting behavior. - hamt: bitfield is big-endian with leading zero bytes stripped, bits are consumed most significant first, prefix width is the hex width of fanout-1, single-entry sub-shards are collapsed on delete - hamt: sharding threshold refers to the IPIP-0499 profiles both implementations ship, not a 256 KiB-1 MiB range neither uses - file chunks: links carry an empty Name, as every existing DAG does - profiles: describe each profile by its own parameters, not by an implementation's current defaults - test vectors: fix block sizes, leaf codecs, link order, and drawings to match the fixture bytes
lidel
marked this pull request as ready for review
September 7, 2026 19:39
🚀 Build Preview on IPFS ready
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
TLDR: Low risk, cleaning up small inconsistencies to reflect interop and default reality.
I'm merging this to leave spec in consistent state with implementations before Shipyard shuts down at the end of the month.
Text-only change. Five statements in
unixfs.mddid not match the bytes in the linked fixtures or what boxo and helia have written since at least January 2023. No requirement on implementations changes.Corrections
HAMT bitfield byte order and length. Spec said little-endian, exactly
fanout/8bytes. kubo writes a big-endian integer with leading zero bytes stripped, identical from v0.18.1 to master, and helia emits the same bytes1:0101 0001 00 0080+ 31 zero bytesunixfs-spec-bitfield-verification.zip if anyone wants to double-check with hexeditor
iiuc bucket 8 in the first byte rules out little-endian; the growing length rules out a fixed width. All 1066 HAMT blocks in the fixture CARs decode only under this reading2.
Bit consumption order. Spec said least significant bits first. Both implementations take bits from the most significant end of the big-endian murmur3 digest:
470.txt(006e88df…) sits at00/6E470.txtin the 1000-file fixture.Prefix width. Spec said
log2(fanout)/4(2.5 for fanout 1024). Both use the hex width offanout-1: 2 for 256, 3 for 1024.Empty
Nameon file chunk links. Spec said absent. Every multi-block file in the fixtures, and every kubo and helia release checked, writes it present and empty (12 00) on each link3. Omitting it changes the CID.Profiles and sharding threshold. Profiles are described by their own frozen parameters instead of "Kubo defaults through v0.39", so the table cannot go stale. The threshold points at the IPIP-499 profiles (256 KiB in both) instead of a "256 KiB to 1 MiB" range no implementation uses.
Test vectors. Block sizes, leaf codecs, link order, and drawings corrected against the CAR bytes; the 1000-file HAMT fixture is three levels deep, not single-layer.
Verified against GO and JS releases since 2023
A 3-entry HAMT (
hello.txt,470.txt,742.txt, threshold 1 byte) and a 3000-byte file in 1024-byte chunks, CIDv1 with raw leaves. Every version below produced the same root blocks: bafybeibxefk42morfuxkbxm2ksusqftpjbno6fafegldo4s2txxun77v2a and bafybeiez22ewz64uclquy3a4m4jecysq5bjhgzuhyhgvshjll5yw77r4s4.Footnotes
go-bitfield
bitfield.go: bucketiat bytelen-1-i/8,Bytes()drops leading zeros. helia'sdir-sharded.tsreverses its little-endian array before writing; the "little endian" comment there describes the pre-reverse array, likely the source of the spec wording. ↩single-layer-hamt-with-multi-block-files.car (237 blocks, June 2023), pbnode-field-orders.car (2), and the two
dir-over-hamtCARs in the IPIP-499 fixtures (827). Six 1-byte bitfields read the same either way. ↩boxo always assigns
Nameon the link (coding.go) and go-codec-dagpb writes any present field (marshal.go); helia writesName: ''. ↩