[#130] Support SerializedFile v26 (Unity 6.7)

[SkowronskiAndrew]

Summary

Fixes #130.

Unity 6.7 moves the SerializedFile format from version 23 to version 26 through three cumulative
changes, two of which alter how a file has to be read. dump and analyze did not work on 6.7
output at all, and most serialized-file subcommands refused it. This makes all of them work,
while continuing to read every older version.

What changed in the format, from Runtime/Serialize/SerializedFileFormatVersion.h:

v	Effect on a reader
24	TypeTree blobs carry their own TypeTreeFormatVersion (32 and up) instead of the file's. The metadata layout is unchanged.
25	The [SerializeReference] registry becomes a self-delimiting frame in the object's data, ahead of the first field that can hold a reference. No TypeTree node describes it - the first piece of object layout a TypeTree cannot express.
26	The file gains a table of shared sub-TypeTrees, stored once and keyed by content hash, that its types reference in place of repeating a compound.

Only <23, 23 and 26 occur in the wild - 24 and 25 shipped inside the 6.7 alpha - but the
changes are cumulative, so all three matter.

The shared-subtree change is the dangerous one. Such a node has a byte size and no children of its
own, which is exactly how every reader of this API recognises a basic type, so a naive reader
takes a fixed-size compound for a primitive of the same width, drops its fields - PPtrs among them -
and still lands in the right place afterwards. Unity's UnityFileSystemApi therefore now refuses
that node through UFS_GetTypeTreeNodeInfo and serves it through a subtree API instead. That
refusal, the registry frame parser, and the independent TypeTree version all arrived in the native
library in Unity commit e06bfae736e1, whose message says outright that the UnityDataTools side is
a separate PR pinning that library. This is that PR.

Where the layout is documented only in the engine source, this uses the library's entry points
rather than re-implementing it. The exception is SerializedFileDetector, which deliberately parses
metadata in pure C# without the native library, and which gains the v26 metadata changes directly.

Changes

Native library and its C# wrapper

• Requires UnityFileSystemApi version 2 or newer, checked once in UnityFileSystem.Init with
  a message naming what is needed. Older files are read through the same library, so there is
  nothing to gain from a fallback. This branch does not contain that library yet - see Before
  merging.
• DllWrapper gains GetTypeTreeSubtreeNodeInfo, GetTypeTreeRefSubtree and the three
  GetRegistryFrame* entry points, the HasSerializedRefs / IsSharedSubtreeRef node flags, and
  the HigherTypeTreeVersion / RequiresSubtreeApi result codes.
• TypeTreeNode now reads every node through the subtree API, passing a null subtree for the
  tree's own nodes, so one walker covers both sides and there is no second path to keep in step. A
  shared subtree reference resolves its children through GetTypeTreeRefSubtree, and is never
  reported as a basic type. RequiresSubtreeApi throws, so a walk that bypasses TypeTreeNode
  fails loudly rather than silently misreading.
• New ManagedReferenceRegistry models the registry: Version, FrameSize and the entries it
  holds. Versions 1 and 2 are read from their TypeTree nodes into the same entries, so consumers see
  one shape whatever the file version. It is public API of the UnityFileSystem library, which
  AGENTS.md describes as reusable for other analysis; inside this repo the frame path is what the
  three walkers use.

Reading objects

The registry frame sits ahead of the field flagged HasSerializedRefs, in root context only - the
same type tree read as a registry blob is frameless. TextDumperTool, PPtrAndCrcProcessor and
RandomAccessReader each step over it and read it; their walks stay separate because each maintains
its own cursor and does something different with the result, but the frame parsing itself is shared.

• dump prints the registry in the shape the version 1 and 2 dumps already used, so the three stay
  comparable. It also prints a null reference, which only the version 3 format can express.
• analyze walks each instance's data through its own type tree for PPtrs, as it already did for
  versions 1 and 2, and CRCs the frame's header and tables as the raw bytes they are. Note that a
  v23 and a v26 build of the same content will not have equal CRCs; the CRC is a walk-derived
  fingerprint, not a hash of the object's bytes (see CRC of objects with references is not comparable across separate databases #74).

serialized-file

• Supported metadata versions now run to 26. Versions 24 and 25 are accepted too: their metadata
  layout is identical to 23, so refusing a layout we can read would be a failure with nothing behind
  it.
• metadata reports the shared subtree table, and in JSON each subtree's content hash, size and
  whether it is inline. Each type entry also reports typeTreeFormatVersion, the stamp on the blob
  itself - 0 before version 23, 23 for version 23 files, 32 or higher from 6.7. That is the only way
  to see a TypeTree change that no longer bumps the file version. Cross-referenced with Expose TypeTree information for SerializedFile #54, which
  covers exposing TypeTree information properly.

Error reporting

analyze surfaced an unreadable version as Sequence contains no elements (reported in #130 while
testing #128). The version is known before the file is opened, so both analyze and dump now say
which version they found and which is the highest they understand, instead of leaving the native
loader to call it a possibly corrupt file.

Documentation

unity-content-format.md gains a "What changed in Unity 6.7" section under TypeTrees. Because the
docs cover every version UnityDataTool reads, the 6.7 behaviour is described as a difference from
6.6 rather than as the way things are. command-serialized-file.md and command-dump.md follow.

Test data

• TestCommon/Data/PlayerWithTypeTreesV26/ - a 6.7 build of the same project and the same two
  scenes as the existing PlayerWithTypeTrees (which is version 22, not 23). Same content
  either side of the format change, so most of the new tests assert the two agree rather than
  pinning v26 on its own. 933 KB, against 814 KB for its older sibling.
• TestCommon/Data/AssetBundleTypeTreeVariations/v26/managedreferences.bundle - Unity's own v26
  fixture, copied from its test corpus, which the two repos already share a source script with. It
  carries the [SerializeReference] shapes nothing else here reaches: a null reference, a PPtr
  inside a referenced object, a collection of references, a reference nested in another, a
  field-less compound, and a plain int before and after the references - that last pair being what
  catches a reader that does not step over the frame, since it reads them shifted rather than
  failing.
• TestCommon/Data/LegacyFormats/v26format.assets - one file pegged at v26, since LeadingEdge moves
  forward with Unity.

Testing

dotnet test - 821 pass, 0 fail (802 before this branch). The pre-existing tests over v17 to v23
files are the backward-compatibility guard that matters most here, since TypeTreeNode now routes
every node through the subtree API, for old files as much as new.

New coverage: shared subtree resolution and the absence of those nodes in a v22 file; the registry
read through all three of its on-disk versions; the detector's shared subtree table and blob version
stamps across v22, v23 and v26; the registry shapes listed above; and the unsupported-version
message.

Verified by hand against the full 48 MB 6.7 player build, not just the checked-in subset: dump
produces clean output for every serialized file in it, and analyze completes with no failures. The
v22 and v26 builds of level0 produce identical analyze graphs (7 objects, 13 references), and
their dumps differ only by a genuinely new 6.7 field (m_CAHMap) and by PathIDs.

Before merging

• The updated UnityFileSystemApi library is not in this branch. The checked-in library is
  still version 1, which the new Init check rejects, so CI will fail on every platform until
  official Windows, macOS and Linux builds from Unity 6.7 are committed here. Everything above was
  developed and verified against a local Windows debug build of 6000.7.0b2
  (UFS_GetDllVersion = 2), which was deliberately kept out of the commits. The suite should be
  re-run once the official libraries land, since only the Windows debug build has been exercised.
• Issues Support Unity 6.7 content directory analysis (ContentLayout.json v3) #131 (6.7 ContentLayout.json) and Unity 6.7 - Support unity archive format v9 #144 (6.7 archive format) are separate 6.7
  problems, deliberately out of scope here. UnityProjects/LeadingEdge was not rebuilt with
  6.7, because doing so would break ContentLayout and Archive tests for those unrelated reasons; the
  coverage it was wanted for is provided by the fixtures above. The version bump and release also
  wait for the point where all the 6.7 fixes are assembled onto one branch.

🤖 Generated with Claude Code

[SkowronskiAndrew]

AGENT: CI is red on all three platforms for a single expected reason, not a test regression:

System.NotSupportedException : UnityFileSystemApi version 1 is too old; version 2 or newer is
required. Replace the UnityFileSystemApi library shipped beside UnityDataTool with one from
Unity 6.7 or newer.

The checked-in native library is still version 1, and reading v26 needs the shared-subtree and
registry-frame entry points that arrived in version 2. Committing official Windows, macOS and Linux
builds from Unity 6.7 should turn all three green; locally, with a Windows debug build of 6000.7.0b2
in place, the suite is 821/821.

The suite is worth re-running once those land, since only the Windows debug build has been exercised
here.

[SkowronskiAndrew]

Perhaps we should set this to 99 or other much later version, so that there is less churn as the version changes.

Line 199 would still need to change, unless there is a way we can reach the actual maximum supported constant so that there is less maintenance burden for this test.

[SkowronskiAndrew]

Done — 99, and the expected-maximum assertion now reads SerializedFileDetector.MaxMetadataParseVersion instead of a literal, so neither line churns. The constant is public for that reason.

[SkowronskiAndrew]

I don't like having the current supported version hardcoded here, we may neglect to update that when new versions come up. We intend to support newer versions as they arrive with new Unity versions, although there can be a time lag. Users of UnityDataTools will often be on older versions of Unity than the current beta or brand new non-LTS release.
We have a runtime error if we don't support a version so perhaps that is better than trying to keep a documentation up to date,

[SkowronskiAndrew]

Done — the version ceiling is gone from the docs. It now says a file from a newer Unity is refused rather than misread, with a message naming the version found and the highest understood, and points at header as the way to check what you have.

[SkowronskiAndrew]

This is quite technical and there is no example. Its not very clear whether the actual dump output would look different between these two versions, or we are just describing an inner detail about the binary representation of v2 and v3.

The comment in Analyzer/PPtrAndCrcProcessor.cs is a good place to get into technical details of the different supported versions because undertanding that actual parsing code requires some explaination. Meanwhile this is the doc for the user facing dump command.

Here it can be worth mentioning SerializeReference output for dump because that is something interesting that users would see when looking at some MonoBehaviors. But I think having a short example is important, possibly just showing an example of the new expected dump output, e.g. v3 instead of getting into the history.

[SkowronskiAndrew]

Done — replaced the history paragraph with an actual example of the output (the references section for ScriptableObjectWIthSerializeReference, taken from the test data), plus one line each on the two things a user actually sees: two fields sharing a rid, and a null reference showing rid -2. The format history is left to the PPtrAndCrcProcessor comment and unity-content-format.md.

On your question of whether the dump output differs between v2 and v3: the entries are identical, and only two things change — the version number, and that the section is printed before the referencing fields in 6.7 instead of after. That is now one sentence at the end rather than the bulk of it.

[SkowronskiAndrew]

again i think a number like 99 would be better than the next expected version, so we don't have to keep churning this test as new versions arrive.

[SkowronskiAndrew]

AGENT: Agreed on the intent, but 99 does not work here specifically. This test goes through real detection, and TryDetectSerializedFile only accepts a version in 1..50 as a plausible SerializedFile header — a version-99 file is not recognised as a SerializedFile at all, so analyze ignores it and exits 0 rather than failing. (The other test hands TryParseMetadata a header directly and bypasses that, which is why 99 is fine there.)

So this one derives it: SerializedFileDetector.MaxMetadataParseVersion + 1, with both assertions built from the same constant. No churn, and it stays inside the detector's plausible range. Happy to raise that 1..50 bound instead if you would rather both tests read the same way, but it is a heuristic that helps reject non-SerializedFiles, so I left it alone.

[SkowronskiAndrew]

I don't understand this comment. Aren't we forced to implement that layout as a second implementation? Is it saying there is a risk of this drifting?

[SkowronskiAndrew]

AGENT: Fair — the comment was badly worded and said the opposite of what I meant. We are not writing a second implementation: UFS_GetRegistryFrameInfo/Types/Records wrap the engine's own ParseRegistryTableRaw, so the only parser of that layout is Unity's, and this class calls it. The one exception is the 8-byte header (version + byteLength), read directly in GetFrameSize, because a walker that only needs to step over a frame should not pay to parse its tables. Rewritten to say that.