diff --git a/AGENTS.md b/AGENTS.md index c03ff9a..5ad53c7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -126,7 +126,7 @@ The feature libraries each build on UnityBinaryFormat and UnityFileSystem. See t **Handlers**: Type-specific handlers extract specialized properties for Unity object types and populate additional tables. For example Mesh, AnimationClip, Shader, BuildReport, MonoScript. -**Views**: The database schema includes convenient views for seeing the data in useful ways, e.g. `object_view`. See `Documentation/analyzer.md` and `Documentation/addressables-build-reports.md` for schema details. +**Views**: The database schema includes convenient views for seeing the data in useful ways, e.g. `object_view`. See `Documentation/analyzer-schema.md` for the core schema, and `Documentation/contentlayout-database.md`, `Documentation/buildreport.md` and `Documentation/addressables-build-reports.md` for the parts documented on their own pages. CLI entry point is `UnityDataTool/Program.cs` using System.CommandLine. Per-command documentation is in `Documentation/`. @@ -135,9 +135,24 @@ CLI entry point is `UnityDataTool/Program.cs` using System.CommandLine. Per-comm ### Extending Analyze * New Unity types can be added by following the same pattern as the existing types, for example MonoScripts. -* Any database schema change (new or changed tables, views, or columns) must bump `PRAGMA user_version` in `Analyzer/Resources/Init.sql` and extend the version-history comment above it. +* Any database schema change (new or changed tables, views, or columns) must bump `PRAGMA user_version` in `Analyzer/Resources/Init.sql` and add a row to the version table in `Documentation/analyzer-schema.md`. * Analysis of additional file formats could be added, for example AssetBundle manifest files by following the pattern of Addressables build layout files are handled. +#### Commenting the .sql resources + +SQLite only keeps the text of the `CREATE` statement itself in `sqlite_master`, so a comment placed +*above* a statement is discarded and never reaches a produced database. Comments therefore go inside +the statement: + +* One short note on the first line inside the `CREATE TABLE ( ... )` parentheses saying what a row + represents, and a trailing `--` note on a column only where the fact is needed to write a correct + query and is not guessable from the column name (a foreign key, a sentinel like `''`, an option + that leaves the column empty). Views get one purpose line as the first line of the body, after `AS`. +* Full prose belongs in the documentation, not in the `.sql` file. +* `CREATE INDEX` has no body, so the only comments left outside a statement are ones that explain the + code rather than the schema, such as why the ContentLayout indexes are created after population. +* Keep `.sql` comments ASCII-only. + ### Other Extensions The UnityFileSystem API and UnityBinaryFormat parsing can be useful for other analysis. The "dump", "analyze" and "serialized-file" commands can be considered reference examples of how to use those lower level tools. diff --git a/Analyzer/Properties/Resources.resx b/Analyzer/Properties/Resources.resx index 0a616aa..27df1f5 100644 --- a/Analyzer/Properties/Resources.resx +++ b/Analyzer/Properties/Resources.resx @@ -131,10 +131,10 @@ ..\Resources\BuildReport.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8 - ..\Resources\Finalize.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;Windows-1252 + ..\Resources\Finalize.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8 - ..\Resources\Init.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;Windows-1252 + ..\Resources\Init.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8 ..\Resources\Mesh.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8 diff --git a/Analyzer/Resources/AssetBundle.sql b/Analyzer/Resources/AssetBundle.sql index b9351cb..b98763f 100644 --- a/Analyzer/Resources/AssetBundle.sql +++ b/Analyzer/Resources/AssetBundle.sql @@ -1,39 +1,28 @@ --- tables related to the AssetBundle and PreloadData objects - --- Do not confuse the AssetBundle Unity object (the source of much of this data) --- with the archives table, which is general to any Unity Archive. - --- The "assets" that an AssetBundle explicitly exposes: each m_Container entry of the AssetBundle --- object names an object (the addressable/asset name -> object it maps to). Populated only from --- the AssetBundle object, so this table is empty for Player and ContentDirectory builds. --- For scene bundles the entry names the scene and points at the synthetic Scene object (see --- AssetBundleHandler / SerializedFileSQLiteWriter). CREATE TABLE IF NOT EXISTS assetbundle_assets( - object INTEGER, - name TEXT + -- The assets an AssetBundle explicitly exposes: one row per m_Container entry of the + -- AssetBundle object. Empty for Player and ContentDirectory builds, which have no such object. + object INTEGER, -- objects.id; for a scene bundle this is the synthetic Scene object + name TEXT -- the container path the asset is addressed by ); --- object depends on dependency. This table has three sources, only the first of which is truly --- AssetBundle-specific: --- * AssetBundleHandler: an asset's slice of the AssetBundle object's m_PreloadTable. --- * SerializedFileSQLiteWriter: a scene object -> each object in the scene's SerializedFiles. --- * PreloadDataHandler: the PreloadData object's m_Assets. PreloadData is a *separate* Unity --- object (not part of the AssetBundle object) and also exists in Player builds (one per scene --- in its sharedAssetsN.assets, plus one in globalgamemanagers.assets), so this table is NOT --- empty there. Player builds have no scene object, so those rows hang off the PreloadData --- object itself; scene bundles hang them off the synthetic Scene object. CREATE TABLE IF NOT EXISTS preload_dependencies( - object INTEGER, - dependency INTEGER + -- Objects that Unity preloads alongside another object. Populated for AssetBundle and Player + -- builds, but not ContentDirectory builds. See Documentation/analyzer-schema.md. + object INTEGER, -- objects.id: an AssetBundle asset, a synthetic Scene, or a PreloadData object + dependency INTEGER -- objects.id, or dangling_refs.id when the target was not analyzed ); CREATE VIEW IF NOT EXISTS assetbundle_asset_view AS +-- AssetBundle assets with their object columns resolved. Inner join, so an asset whose object was +-- not analyzed is omitted here but still present in assetbundle_assets. SELECT a.name AS asset_name, o.* FROM assetbundle_assets a INNER JOIN object_view o ON o.id = a.object; CREATE VIEW IF NOT EXISTS preload_dependencies_view AS +-- Preload dependencies of AssetBundle assets and scenes, with both sides resolved. Narrower than +-- the table: Player-build rows and dangling dependencies drop out of the inner joins. SELECT a.id, a.asset_name, a.archive, a.type, od.id dep_id, od.archive dep_archive, od.name dep_name, od.type dep_type FROM assetbundle_asset_view a INNER JOIN preload_dependencies d ON a.id = d.object diff --git a/Analyzer/Resources/ContentLayout.sql b/Analyzer/Resources/ContentLayout.sql index 1896e4a..ef322b7 100644 --- a/Analyzer/Resources/ContentLayout.sql +++ b/Analyzer/Resources/ContentLayout.sql @@ -1,8 +1,7 @@ --- Identity of the imported ContentLayout.json (see Documentation/contentlayout.md). The --- content_layout* tables are only created when a ContentLayout.json is part of the analyzed --- input. A single layout per database is supported. CREATE TABLE IF NOT EXISTS content_layout ( + -- Identity of the imported ContentLayout.json. The content_layout* tables exist only when a + -- layout was part of the analyzed input. See Documentation/contentlayout-database.md. id INTEGER, -- always 0 (single layout per database) name TEXT, -- path of the imported ContentLayout.json version INTEGER, -- schema version of the json file diff --git a/Analyzer/Resources/ContentLayoutArtifactReferences.sql b/Analyzer/Resources/ContentLayoutArtifactReferences.sql index cf63fdd..fc55c36 100644 --- a/Analyzer/Resources/ContentLayoutArtifactReferences.sql +++ b/Analyzer/Resources/ContentLayoutArtifactReferences.sql @@ -1,9 +1,8 @@ --- Direct references between binary artifacts (ArtifactReferences in the json), e.g. a --- serialized file referencing its .resS/.resource data files. References that go through a --- loadable are not included, and the graph is never cyclical. References to other serialized --- files are not recorded here either; those are in content_layout_serialized_file_dependencies. CREATE TABLE IF NOT EXISTS content_layout_artifact_references ( + -- Direct references between binary artifacts, e.g. a content file to its .resS/.resource data + -- files. Never cyclical. Content-file-to-content-file edges live in + -- content_layout_serialized_file_dependencies instead. artifact_index INTEGER, -- references content_layout_binary_artifacts.artifact_index referenced_artifact_index INTEGER, PRIMARY KEY (artifact_index, referenced_artifact_index) diff --git a/Analyzer/Resources/ContentLayoutBinaryArtifacts.sql b/Analyzer/Resources/ContentLayoutBinaryArtifacts.sql index 9fa21f0..f1ad455 100644 --- a/Analyzer/Resources/ContentLayoutBinaryArtifacts.sql +++ b/Analyzer/Resources/ContentLayoutBinaryArtifacts.sql @@ -1,11 +1,9 @@ --- The artifacts that make up the build output (BinaryArtifacts in the json): the serialized --- files themselves plus the data files they use (.resS, .resource) and the manifest. This is the --- standard place to find artifact sizes. When stored as a file the filename is the content hash --- plus an extension derived from the category (content_layout_binary_artifacts_view adds it). CREATE TABLE IF NOT EXISTS content_layout_binary_artifacts ( + -- Every artifact making up the build output: the content files plus the data files they use + -- (.resS, .resource) and the manifest. The standard place to find artifact sizes. artifact_index INTEGER, - content_hash TEXT, + content_hash TEXT, -- the on-disk filename is content_hash plus an extension from category category TEXT, -- 'texture' | 'mesh' | 'audio' | 'video' | 'contentfile' | 'manifest' size INTEGER, PRIMARY KEY (artifact_index) diff --git a/Analyzer/Resources/ContentLayoutIndexes.sql b/Analyzer/Resources/ContentLayoutIndexes.sql index 78dbf94..b4b0b1d 100644 --- a/Analyzer/Resources/ContentLayoutIndexes.sql +++ b/Analyzer/Resources/ContentLayoutIndexes.sql @@ -1,6 +1,4 @@ --- Created after the content_layout tables are populated, so inserts stay fast for very large --- layouts. The content_hash and asset_path indexes carry the views; the rest serve reverse --- lookups ("who depends on X", "which loadables live in file Y"). +-- Created after the content_layout tables are populated so inserts stay fast for very large layouts. CREATE INDEX content_layout_binary_artifacts_content_hash ON content_layout_binary_artifacts(content_hash); CREATE INDEX content_layout_source_assets_asset_path ON content_layout_source_assets(asset_path); CREATE INDEX content_layout_source_assets_file ON content_layout_source_assets(serialized_file_index); diff --git a/Analyzer/Resources/ContentLayoutLoadableDependencies.sql b/Analyzer/Resources/ContentLayoutLoadableDependencies.sql index 4293a4f..850e554 100644 --- a/Analyzer/Resources/ContentLayoutLoadableDependencies.sql +++ b/Analyzer/Resources/ContentLayoutLoadableDependencies.sql @@ -1,6 +1,6 @@ --- Loadable objects referenced from each serialized file (LoadableDependencies in the json). CREATE TABLE IF NOT EXISTS content_layout_loadable_dependencies ( + -- The loadable objects each content file references. serialized_file_index INTEGER, -- references content_layout_serialized_files.file_index object_id_hash TEXT -- references content_layout_loadable_objects.object_id_hash ); diff --git a/Analyzer/Resources/ContentLayoutLoadableObjects.sql b/Analyzer/Resources/ContentLayoutLoadableObjects.sql index 179465b..e34447b 100644 --- a/Analyzer/Resources/ContentLayoutLoadableObjects.sql +++ b/Analyzer/Resources/ContentLayoutLoadableObjects.sql @@ -1,17 +1,14 @@ --- The objects that can be loaded on demand (LoadableObjectIds in the json), identified --- independently of the serialized file that contains them. Also records where each one came from --- in the source project. The json's top-level RootAssets list is folded into the is_root_asset --- flag. serialized_file_index is NULL when the object was dropped from the build (json value -1, --- e.g. server build shader references). CREATE TABLE IF NOT EXISTS content_layout_loadable_objects ( + -- The objects that can be loaded on demand, identified independently of the content file that + -- holds them, plus where each came from in the source project. object_id_hash TEXT, -- hash of GUID, LFID and identifier_type guid TEXT, -- AssetDatabase GUID of the source asset asset_path TEXT, lfid INTEGER, -- local file id of the object in the source asset identifier_type INTEGER, - serialized_file_index INTEGER, -- references content_layout_serialized_files.file_index, or NULL - output_lfid INTEGER, -- local file id of the object in its output serialized file + serialized_file_index INTEGER, -- content_layout_serialized_files.file_index; NULL if dropped from the build + output_lfid INTEGER, -- local file id of the object in its output content file is_root_asset INTEGER, PRIMARY KEY (object_id_hash) ); diff --git a/Analyzer/Resources/ContentLayoutLoadableSceneDependencies.sql b/Analyzer/Resources/ContentLayoutLoadableSceneDependencies.sql index 84c2eac..6ee3d9d 100644 --- a/Analyzer/Resources/ContentLayoutLoadableSceneDependencies.sql +++ b/Analyzer/Resources/ContentLayoutLoadableSceneDependencies.sql @@ -1,6 +1,6 @@ --- Scenes referenced from each serialized file (LoadableSceneDependencies in the json). CREATE TABLE IF NOT EXISTS content_layout_loadable_scene_dependencies ( + -- The scenes each content file references. serialized_file_index INTEGER, -- references content_layout_serialized_files.file_index scene_path TEXT -- matches content_layout_loadable_scenes.path ); diff --git a/Analyzer/Resources/ContentLayoutLoadableScenes.sql b/Analyzer/Resources/ContentLayoutLoadableScenes.sql index 7148b88..c01aa30 100644 --- a/Analyzer/Resources/ContentLayoutLoadableScenes.sql +++ b/Analyzer/Resources/ContentLayoutLoadableScenes.sql @@ -1,8 +1,8 @@ --- The scenes exposed as loadable in the build (LoadableSceneIds in the json). CREATE TABLE IF NOT EXISTS content_layout_loadable_scenes ( + -- The scenes exposed as loadable in the build. guid TEXT, path TEXT, - serialized_file_index INTEGER, -- references content_layout_serialized_files.file_index, or NULL + serialized_file_index INTEGER, -- content_layout_serialized_files.file_index; NULL if dropped from the build PRIMARY KEY (guid) ); diff --git a/Analyzer/Resources/ContentLayoutSerializedFileDependencies.sql b/Analyzer/Resources/ContentLayoutSerializedFileDependencies.sql index d71bdb1..093d2c6 100644 --- a/Analyzer/Resources/ContentLayoutSerializedFileDependencies.sql +++ b/Analyzer/Resources/ContentLayoutSerializedFileDependencies.sql @@ -1,11 +1,8 @@ --- File-to-file dependency edges (SerializedFileDependencies in the json): the other serialized --- files that must be loaded before this one. position preserves the array order, which is --- significant: a PPtr's m_FileID inside the file resolves positionally through this list (see --- Documentation/contentdirectory-format.md). CREATE TABLE IF NOT EXISTS content_layout_serialized_file_dependencies ( + -- File-to-file dependency edges: the other content files that must be loaded before this one. serialized_file_index INTEGER, -- references content_layout_serialized_files.file_index - position INTEGER, -- 1-based, matching the external reference table / m_FileID index + position INTEGER, -- 1-based; a PPtr's m_FileID resolves positionally through this list dependency_index INTEGER, -- references content_layout_serialized_files.file_index PRIMARY KEY (serialized_file_index, position) ); diff --git a/Analyzer/Resources/ContentLayoutSerializedFiles.sql b/Analyzer/Resources/ContentLayoutSerializedFiles.sql index 3bf7b0b..5af0b21 100644 --- a/Analyzer/Resources/ContentLayoutSerializedFiles.sql +++ b/Analyzer/Resources/ContentLayoutSerializedFiles.sql @@ -1,15 +1,11 @@ --- One row per entry in the layout's SerializedFiles array: the serialized files (.cf Content --- Files) that the ContentDirectory build produced. file_index is the array index from the json --- and is how the other content_layout tables reference a file. --- serialized_file links to the core serialized_files table so layout data joins directly with --- analyzed objects and references; it is NULL when the analyzed input did not include the build --- content (e.g. a layout-only analyze) or for built-in entries. CREATE TABLE IF NOT EXISTS content_layout_serialized_files ( - file_index INTEGER, + -- The content files (.cf) the ContentDirectory build produced, one row per entry in the + -- layout's SerializedFiles array. + file_index INTEGER, -- the json array index; how the other content_layout tables name a file cfid TEXT, -- symbolic .cfid reference string (for built-ins: the built-in path) is_builtin INTEGER, content_hash TEXT, -- NULL for built-ins; the filename is content_hash || '.cf' - serialized_file INTEGER, -- references serialized_files.id, or NULL + serialized_file INTEGER, -- serialized_files.id; NULL for built-ins and for a layout-only analyze PRIMARY KEY (file_index) ); diff --git a/Analyzer/Resources/ContentLayoutSourceAssets.sql b/Analyzer/Resources/ContentLayoutSourceAssets.sql index b50458b..ac33c85 100644 --- a/Analyzer/Resources/ContentLayoutSourceAssets.sql +++ b/Analyzer/Resources/ContentLayoutSourceAssets.sql @@ -1,7 +1,7 @@ --- The source assets included in each serialized file (SourceAssets in the json). The same asset --- path can appear in more than one file (e.g. an FBX split into multiple output files). CREATE TABLE IF NOT EXISTS content_layout_source_assets ( + -- The source assets built into each content file. The same asset path can appear in more than + -- one file, so this is many-to-many. serialized_file_index INTEGER, -- references content_layout_serialized_files.file_index asset_path TEXT ); diff --git a/Analyzer/Resources/ContentLayoutViews.sql b/Analyzer/Resources/ContentLayoutViews.sql index 134a356..2db54f6 100644 --- a/Analyzer/Resources/ContentLayoutViews.sql +++ b/Analyzer/Resources/ContentLayoutViews.sql @@ -1,10 +1,6 @@ --- Convenience views over the content_layout tables. Created together with the tables (only when --- a ContentLayout.json is imported). See Documentation/contentlayout-database.md for the schema --- reference. - --- One row per layout serialized file with the derived filename, artifact size, and core-table link. --- Built-in entries have no file on disk, so their path (cfid) is shown as the filename. CREATE VIEW IF NOT EXISTS content_layout_serialized_files_view AS +-- One row per layout content file with its derived filename, artifact size and core-table link. +-- Built-in entries have no file on disk, so their path (cfid) is shown as the filename. SELECT f.file_index, f.cfid, f.is_builtin, CASE WHEN f.is_builtin = 1 THEN f.cfid ELSE f.content_hash || '.cf' END AS filename, ba.size, @@ -14,24 +10,22 @@ FROM content_layout_serialized_files f LEFT JOIN content_layout_binary_artifacts ba ON ba.content_hash = f.content_hash AND ba.category = 'contentfile' LEFT JOIN serialized_files sf ON sf.id = f.serialized_file; --- Source asset -> the file(s) it was built into. Built-in entries have no source assets, so no --- filename fallback is needed here. CREATE VIEW IF NOT EXISTS content_layout_source_assets_view AS +-- Source asset to the content file(s) it was built into. SELECT s.asset_path, f.file_index, f.content_hash || '.cf' AS filename, f.serialized_file FROM content_layout_source_assets s INNER JOIN content_layout_serialized_files f ON f.file_index = s.serialized_file_index; --- File-to-file dependency edges with names resolved on both sides (via the files view, so --- dependencies on built-in entries show their path instead of a NULL filename). CREATE VIEW IF NOT EXISTS content_layout_serialized_file_dependencies_view AS +-- File-to-file dependency edges with filenames resolved on both sides. SELECT d.serialized_file_index, src.filename, d.position, d.dependency_index, dep.filename AS dependency_filename, dep.cfid AS dependency_cfid FROM content_layout_serialized_file_dependencies d INNER JOIN content_layout_serialized_files_view src ON src.file_index = d.serialized_file_index INNER JOIN content_layout_serialized_files_view dep ON dep.file_index = d.dependency_index; --- Loadables resolved to their analyzed object (object columns are NULL in a layout-only database). CREATE VIEW IF NOT EXISTS content_layout_loadable_objects_view AS +-- Loadables resolved to their analyzed object. The object columns are NULL in a layout-only database. SELECT l.object_id_hash, l.guid, l.asset_path, l.lfid, l.is_root_asset, f.content_hash || '.cf' AS filename, o.id AS object, t.name AS type, o.name, o.size @@ -40,8 +34,8 @@ LEFT JOIN content_layout_serialized_files f ON f.file_index = l.serialized_file_ LEFT JOIN objects o ON o.serialized_file = f.serialized_file AND o.object_id = l.output_lfid LEFT JOIN types t ON t.id = o.type; --- Artifacts with their derived on-disk filename (extension based on the category). CREATE VIEW IF NOT EXISTS content_layout_binary_artifacts_view AS +-- Artifacts with their on-disk filename derived from the content hash and category. SELECT artifact_index, content_hash, category, size, content_hash || CASE category @@ -52,8 +46,8 @@ SELECT artifact_index, content_hash, category, size, END AS filename FROM content_layout_binary_artifacts; --- The data files (.resS/.resource) each serialized file uses, derived from the artifact graph. CREATE VIEW IF NOT EXISTS content_layout_resource_files_view AS +-- The data files (.resS/.resource) each content file uses, derived from the artifact graph. SELECT f.file_index, f.content_hash || '.cf' AS filename, ra.category, rav.filename AS data_filename, ra.size FROM content_layout_serialized_files f diff --git a/Analyzer/Resources/ContentSummary.sql b/Analyzer/Resources/ContentSummary.sql index 9c684f0..1087677 100644 --- a/Analyzer/Resources/ContentSummary.sql +++ b/Analyzer/Resources/ContentSummary.sql @@ -31,10 +31,9 @@ CREATE TABLE IF NOT EXISTS build_report_content_asset_stats( FOREIGN KEY (content_summary_id) REFERENCES build_report_content_summary(id) ); --- Cross-build statistics with the owning BuildReport resolved. build_report_id is the id of the --- BuildReport object (type 1125) in the same serialized file as the ContentSummary; it is not --- stored on the table, so this view computes it the same way build_report_packed_assets_view does. CREATE VIEW build_report_content_summary_view AS +-- Cross-build content statistics with the owning BuildReport resolved, so one build can be +-- selected by build_report_id. SELECT cs.id AS content_summary_id, br_obj.id AS build_report_id, @@ -52,9 +51,9 @@ INNER JOIN objects o ON cs.id = o.id INNER JOIN serialized_files sf ON o.serialized_file = sf.id LEFT JOIN objects br_obj ON o.serialized_file = br_obj.serialized_file AND br_obj.type = 1125; --- Per-type statistics with the type name resolved (from TypeIdRegistry or TypeTree analysis) and --- the owning BuildReport, so a single build's type breakdown can be selected by build_report_id. CREATE VIEW build_report_content_type_stats_view AS +-- Per-type statistics with the type name and the owning BuildReport resolved, so a single build's +-- type breakdown can be selected by build_report_id. SELECT cs.id AS content_summary_id, br_obj.id AS build_report_id, diff --git a/Analyzer/Resources/Init.sql b/Analyzer/Resources/Init.sql index 666ab77..ef1ea5c 100644 --- a/Analyzer/Resources/Init.sql +++ b/Analyzer/Resources/Init.sql @@ -1,121 +1,89 @@ CREATE TABLE IF NOT EXISTS types ( - id INTEGER, + -- Unity type names, referenced by objects.type. + id INTEGER, -- Unity class id; -1 is the synthetic Scene type analyze adds for scenes name TEXT, PRIMARY KEY (id) ); --- Describes a unity archive that contains serialized files and other built content. --- A common use of the unity archive is for AssetBundles but it can also be used for --- Player, Content Archive and ContentDirectory builds. --- name is UNIQUE (case-sensitive, matching the name on the file system): analyze only supports a --- single build, so two archives with the same name would make queries ambiguous. A duplicate is --- caught in code and reported (see AnalyzeDuplicateException); the constraint is the durable --- backstop for that invariant. CREATE TABLE IF NOT EXISTS archives ( + -- One row per Unity Archive: AssetBundles, and the archives inside Player and + -- ContentDirectory builds. Full reference: Documentation/analyzer-schema.md. id INTEGER, - name TEXT, + name TEXT, -- UNIQUE: analyze covers a single build, so names must not collide file_size INTEGER, PRIMARY KEY (id), UNIQUE (name) ); --- One row per SerializedFile encountered during analysis. The name is often a technical, --- hash-based string rather than a readable path, because that is how the file is named on disk: --- * Regular AssetBundles: "CAB-". This is MD4, not Unity's --- Hash128 (spooky hash), and the optional AssetBundle-filename hash is not part of it. --- * Scene bundles vary by build pipeline: BuildPipeline.BuildAssetBundles uses --- "BuildPlayer-"; the Scriptable Build Pipeline / Addressables uses --- "CAB-"; the Multi-Process Build Pipeline uses "CAB-". --- * Player builds name scenes "level0", "level1", ... in scene-list order. --- archive references the row from archives table of the unity archive containing this file, --- or is '' when the file is not inside an unity archive; object_view turns that '' into NULL. CREATE TABLE IF NOT EXISTS serialized_files ( + -- One row per SerializedFile analyzed. id INTEGER, - archive INTEGER, - name TEXT, + archive INTEGER, -- archives.id, or '' when the file is not inside an archive + name TEXT, -- on-disk name, usually hash-based: CAB-, BuildPlayer-, levelN PRIMARY KEY (id) ); --- Records information about each Unity Object discovered in the analyze process. --- id - unique id for the object (assigned while populating the database, this value does not exist in the serialized content) --- object_id - Local file id for the object, serialized as m_PathID in the object references (PPTRs). signed 64 bit. --- This is unique within a serialized file, but not across files. --- type - references the row in the types table. --- name - the Object.name property. In many case this is empty. --- game_object - only applies to components in a game object hierarchy, otherwise it is not set. --- crc32 - the CRC of the serialized state of the object, including any external .resS or .resource content. --- Useful for comparing builds to detect differences CREATE TABLE IF NOT EXISTS objects ( - id INTEGER, - object_id INTEGER, - serialized_file INTEGER, - type INTEGER, - name TEXT, - game_object INTEGER, - size INTEGER, - crc32 INTEGER, + -- One row per Unity object found. Query object_view for the resolved type, file and + -- archive names. + id INTEGER, -- analyzer-assigned; does not exist in the serialized data + object_id INTEGER, -- Unity local file id (m_PathID), signed 64-bit, unique only within its file + serialized_file INTEGER, -- serialized_files.id + type INTEGER, -- types.id + name TEXT, -- Object.name; often empty + game_object INTEGER, -- objects.id of the owning GameObject; '' when not a component + size INTEGER, -- includes external .resS / .resource bytes + crc32 INTEGER, -- CRC of the serialized state incl. stream data; 0 when --skip-crc was used PRIMARY KEY (id) ); --- Deduplicated lookup tables for the strings referenced by the refs table. --- refs stores ids into these instead of repeating the strings on every row. CREATE TABLE IF NOT EXISTS property_names ( + -- Distinct property paths referenced by refs.property_path. Join through refs_view. id INTEGER PRIMARY KEY, - name TEXT + name TEXT -- e.g. m_Shader, m_Materials[0] ); CREATE TABLE IF NOT EXISTS property_types ( + -- Distinct referenced type names referenced by refs.property_type. Join through refs_view. id INTEGER PRIMARY KEY, - name TEXT + name TEXT -- e.g. Texture2D, MonoScript ); --- Tracks all references between Unity Objects (e.g. PPTRs) --- These references can exist between objects together inside the same serialized file, or they can --- span between serialized files. CREATE TABLE IF NOT EXISTS refs ( - object INTEGER, - referenced_object INTEGER, - property_path INTEGER, - property_type INTEGER + -- References between Unity objects (PPtrs), within and across serialized files. + -- Empty when analyze was run with --skip-references. + object INTEGER, -- objects.id + referenced_object INTEGER, -- objects.id, or dangling_refs.id when the target was not analyzed + property_path INTEGER, -- property_names.id + property_type INTEGER -- property_types.id ); --- One row per referenced object that analyze assigned an id to but never wrote an objects row --- for (its serialized file was not part of the analyzed input). Written after all files are --- processed, once we can tell which assigned ids never became objects. Common causes: analyzing a --- partial set of AssetBundles, references into "unity default resources" (shipped without --- TypeTrees), or a ContentDirectory build analyzed without ContentLayout.json. --- Columns mirror the objects table so every object id resolves to exactly one of objects or --- dangling_refs: --- id - the analyzer object id (no row in objects has this id). --- object_id - the target's local file id (LFID / m_PathID) within its serialized file. --- serialized_file - references the serialized_files row of the (un-analyzed) file the target --- lives in. That row has archive NULL and no objects of its own. CREATE TABLE IF NOT EXISTS dangling_refs ( - id INTEGER, - object_id INTEGER, - serialized_file INTEGER, + -- Reference targets that were never analyzed because their serialized file was not part of the + -- input. Every id used by refs resolves to exactly one of objects or dangling_refs. + id INTEGER, -- the assigned object id; no objects row has it + object_id INTEGER, -- the target's local file id (m_PathID) within its file + serialized_file INTEGER, -- serialized_files.id of the un-analyzed file; it has no objects PRIMARY KEY (id) ); --- Resolves the property_path and property_type ids in the refs table to their string values. CREATE VIEW refs_view AS +-- refs with the property_path and property_type ids resolved to their strings. SELECT r.object, r.referenced_object, pn.name AS property_path, pt.name AS property_type FROM refs r INNER JOIN property_names pn ON r.property_path = pn.id INNER JOIN property_types pt ON r.property_type = pt.id; --- Resolves dangling_refs to the source object(s) that reference each missing target, one row per --- (referencing object -> dangling target) reference. Not populated when analyze is run with --- --skip-references (neither refs nor dangling_refs are populated in that mode). CREATE VIEW dangling_refs_view AS +-- One row per reference to an un-analyzed target. Empty with --skip-references. SELECT r.object AS source_id, src_sf.name AS source_serialized_file, @@ -134,6 +102,7 @@ LEFT JOIN property_names pn ON r.property_path = pn.id LEFT JOIN property_types pt ON r.property_type = pt.id; CREATE VIEW object_view AS +-- The main view: every analyzed object with its type, file and archive names resolved. Start here. SELECT o.id, o.object_id, ab.name AS archive, sf.name AS serialized_file, t.name AS type, o.name, o.game_object, o.size, CASE WHEN size < 1024 THEN printf('%!5.1f B', size * 1.0) @@ -147,6 +116,7 @@ INNER JOIN serialized_files sf ON o.serialized_file = sf.id LEFT JOIN archives ab ON sf.archive = ab.id; CREATE VIEW view_breakdown_by_type AS +-- Object count and total size per type, largest first. SELECT *, CASE WHEN byte_size < 1024 THEN printf('%!5.1f B', byte_size * 1.0) @@ -161,6 +131,8 @@ GROUP BY type ORDER BY byte_size DESC, count DESC); CREATE VIEW view_potential_duplicates AS +-- Objects with identical name, type, size and crc32 found in more than one file. --skip-crc sets +-- every crc32 to 0, which makes this view produce many false positives. SELECT COUNT(name) AS instances, name, type, CASE WHEN sum(size) < 1024 THEN printf('%!5.1f B', sum(size) * 1.0) @@ -178,6 +150,7 @@ HAVING instances > 1 ORDER BY size DESC, instances DESC; CREATE VIEW view_material_shader_refs AS +-- Each Material and the Shader it references, with the AssetBundle path when there is one. SELECT m.id material_id, m.name material_name, a.name material_path, m.archive material_archive, s.id shader_id, s.name shader_name, s.archive shader_archive FROM object_view m INNER JOIN refs_view r ON m.id = r.object AND r.property_path = 'm_Shader' @@ -185,6 +158,7 @@ INNER JOIN object_view s ON r.referenced_object = s.id LEFT JOIN assetbundle_assets a ON m.id = a.object; CREATE VIEW view_material_texture_refs AS +-- Each Material and the Textures it references, with the AssetBundle path when there is one. SELECT m.id material_id, m.name material_name, a.name material_path, m.archive material_archive, t.id texture_id, t.name texture_name, t.archive texture_archive FROM object_view m INNER JOIN refs_view r ON r.object = m.id AND property_type = 'Texture' @@ -192,20 +166,8 @@ INNER JOIN object_view t ON r.referenced_object = t.id LEFT JOIN assetbundle_assets a ON m.id = a.object WHERE m.type = 'Material'; --- Special-case type value for the fake Scene object that is sometimes inserted into the object table, --- see SerializedFileSQLiteWriter for details INSERT INTO types (id, name) VALUES (-1, 'Scene'); --- Database schema version. Bump on any schema change so the version records which schema a given --- analyze.db was produced with. Commands that read an existing database (currently only find-refs) --- can compare against it to give a clean error instead of failing on a missing table or column. --- 1 = normalized refs table (issue #44); 2 = renamed assets/asset_dependencies tables to --- assetbundle_assets/preload_dependencies (issue #82); 3 = renamed asset_bundles table to archives --- and the asset_bundle column/alias to archive (issue #68); 4 = build_report_packed_asset_contents_view --- type column changed from numeric id to type name (issue #55); 5 = added dangling_refs table/view --- (issue #85); 6 = archives.name is unique (issue #51); 7 = Unity 6.6 build_reports columns and --- build_report_content_* tables (issue #107), asset_name/asset_extension columns on --- build_report_source_assets (issue #110); databases produced before versioning report 0. PRAGMA user_version = 7; PRAGMA synchronous = OFF; diff --git a/Analyzer/SQLite/Commands/SerializedFile/AddArchive.cs b/Analyzer/SQLite/Commands/SerializedFile/AddArchive.cs index d0d9ea6..12afe36 100644 --- a/Analyzer/SQLite/Commands/SerializedFile/AddArchive.cs +++ b/Analyzer/SQLite/Commands/SerializedFile/AddArchive.cs @@ -3,16 +3,7 @@ namespace UnityDataTools.Analyzer.SQLite.Commands.SerializedFile { - /* TABLE DEFINITION: - create table archives - ( - id INTEGER, - name TEXT, - file_size INTEGER, - PRIMARY KEY (id), - UNIQUE (name) - ); - */ + // Table definition: Analyzer/Resources/Init.sql internal class AddArchive : AbstractCommand { protected override string TableName => "archives"; diff --git a/Analyzer/SQLite/Commands/SerializedFile/AddDanglingRef.cs b/Analyzer/SQLite/Commands/SerializedFile/AddDanglingRef.cs index 1af0907..c1a4a89 100644 --- a/Analyzer/SQLite/Commands/SerializedFile/AddDanglingRef.cs +++ b/Analyzer/SQLite/Commands/SerializedFile/AddDanglingRef.cs @@ -4,15 +4,7 @@ namespace UnityDataTools.Analyzer.SQLite.Commands.SerializedFile { - /* TABLE DEFINITION: - create table dangling_refs - ( - id INTEGER, - object_id INTEGER, - serialized_file INTEGER, - PRIMARY KEY (id) - ); - */ + // Table definition: Analyzer/Resources/Init.sql internal class AddDanglingRef : AbstractCommand { protected override string TableName => "dangling_refs"; diff --git a/Analyzer/SQLite/Commands/SerializedFile/AddObject.cs b/Analyzer/SQLite/Commands/SerializedFile/AddObject.cs index 9ff1034..b9e2b8a 100644 --- a/Analyzer/SQLite/Commands/SerializedFile/AddObject.cs +++ b/Analyzer/SQLite/Commands/SerializedFile/AddObject.cs @@ -4,20 +4,7 @@ namespace UnityDataTools.Analyzer.SQLite.Commands.SerializedFile { - /* TABLE DEFINITION: - create table objects - ( - id INTEGER, - object_id INTEGER, - serialized_file INTEGER, - type INTEGER, - name TEXT, - game_object INTEGER, - size INTEGER, - crc32 INTEGER, - PRIMARY KEY (id) - ); - */ + // Table definition: Analyzer/Resources/Init.sql internal class AddObject : AbstractCommand { protected override string TableName => "objects"; diff --git a/Analyzer/SQLite/Commands/SerializedFile/AddPreloadDependency.cs b/Analyzer/SQLite/Commands/SerializedFile/AddPreloadDependency.cs index 9dc9e28..744e1df 100644 --- a/Analyzer/SQLite/Commands/SerializedFile/AddPreloadDependency.cs +++ b/Analyzer/SQLite/Commands/SerializedFile/AddPreloadDependency.cs @@ -4,14 +4,7 @@ namespace UnityDataTools.Analyzer.SQLite.Commands.SerializedFile { - /* TABLE DEFINITION: - create table preload_dependencies - ( - object INTEGER, - dependency INTEGER, - PRIMARY KEY (object, dependency) - ); - */ + // Table definition: Analyzer/Resources/AssetBundle.sql internal class AddPreloadDependency : AbstractCommand { protected override string TableName => "preload_dependencies"; diff --git a/Analyzer/SQLite/Commands/SerializedFile/AddPropertyName.cs b/Analyzer/SQLite/Commands/SerializedFile/AddPropertyName.cs index 8838113..c01289e 100644 --- a/Analyzer/SQLite/Commands/SerializedFile/AddPropertyName.cs +++ b/Analyzer/SQLite/Commands/SerializedFile/AddPropertyName.cs @@ -4,14 +4,7 @@ namespace UnityDataTools.Analyzer.SQLite.Commands.SerializedFile { - /* TABLE DEFINITION: - create table property_names - ( - id INTEGER, - name TEXT, - PRIMARY KEY (id) - ); - */ + // Table definition: Analyzer/Resources/Init.sql internal class AddPropertyName : AbstractCommand { protected override string TableName => "property_names"; diff --git a/Analyzer/SQLite/Commands/SerializedFile/AddPropertyType.cs b/Analyzer/SQLite/Commands/SerializedFile/AddPropertyType.cs index 04119c7..37c5b9d 100644 --- a/Analyzer/SQLite/Commands/SerializedFile/AddPropertyType.cs +++ b/Analyzer/SQLite/Commands/SerializedFile/AddPropertyType.cs @@ -4,14 +4,7 @@ namespace UnityDataTools.Analyzer.SQLite.Commands.SerializedFile { - /* TABLE DEFINITION: - create table property_types - ( - id INTEGER, - name TEXT, - PRIMARY KEY (id) - ); - */ + // Table definition: Analyzer/Resources/Init.sql internal class AddPropertyType : AbstractCommand { protected override string TableName => "property_types"; diff --git a/Analyzer/SQLite/Commands/SerializedFile/AddReference.cs b/Analyzer/SQLite/Commands/SerializedFile/AddReference.cs index 94f1bd3..24cc342 100644 --- a/Analyzer/SQLite/Commands/SerializedFile/AddReference.cs +++ b/Analyzer/SQLite/Commands/SerializedFile/AddReference.cs @@ -4,15 +4,7 @@ namespace UnityDataTools.Analyzer.SQLite.Commands.SerializedFile { - /* TABLE DEFINITION: - create table refs - ( - object INTEGER, - referenced_object INTEGER, - property_path INTEGER, -- id into property_names - property_type INTEGER -- id into property_types - ); - */ + // Table definition: Analyzer/Resources/Init.sql internal class AddReference : AbstractCommand { protected override string TableName => "refs"; diff --git a/Analyzer/SQLite/Commands/SerializedFile/AddSerializedFile.cs b/Analyzer/SQLite/Commands/SerializedFile/AddSerializedFile.cs index 40c4ba6..c701ce0 100644 --- a/Analyzer/SQLite/Commands/SerializedFile/AddSerializedFile.cs +++ b/Analyzer/SQLite/Commands/SerializedFile/AddSerializedFile.cs @@ -4,15 +4,7 @@ namespace UnityDataTools.Analyzer.SQLite.Commands.SerializedFile { - /* TABLE DEFINITION: - create table serialized_files - ( - id INTEGER, - archive INTEGER, - name TEXT, - PRIMARY KEY (id) - ); - */ + // Table definition: Analyzer/Resources/Init.sql internal class AddSerializedFile : AbstractCommand { protected override string TableName => "serialized_files"; diff --git a/Analyzer/SQLite/Commands/SerializedFile/AddType.cs b/Analyzer/SQLite/Commands/SerializedFile/AddType.cs index 8f94c86..2a7ded1 100644 --- a/Analyzer/SQLite/Commands/SerializedFile/AddType.cs +++ b/Analyzer/SQLite/Commands/SerializedFile/AddType.cs @@ -4,14 +4,7 @@ namespace UnityDataTools.Analyzer.SQLite.Commands.SerializedFile { - /* TABLE DEFINITION: - create table types - ( - id INTEGER, - name TEXT, - PRIMARY KEY (id) - ); - */ + // Table definition: Analyzer/Resources/Init.sql internal class AddType : AbstractCommand { protected override string TableName => "types"; diff --git a/Documentation/agent-guide.md b/Documentation/agent-guide.md index 0e0f63a..c18fe81 100644 --- a/Documentation/agent-guide.md +++ b/Documentation/agent-guide.md @@ -35,8 +35,8 @@ pasted (or linked) into an agent's context, and it is just as useful for humans sqlite3 Analysis.db "SELECT name FROM sqlite_master WHERE type = 'view' ORDER BY name;" ``` - The full schema is documented in the [Analyzer database reference](analyzer.md); worked example - queries are in [Example usage of Analyze](analyze-examples.md). + The full schema is documented in the [Analyzer database schema](analyzer-schema.md); worked + example queries are in [Example usage of Analyze](analyze-examples.md). 4. **Drill down into specific files and objects.** Once a query has identified something interesting, the other commands show the actual content: @@ -162,15 +162,16 @@ context. Dump the schema of your database into a text file and attach it before sqlite3 Analysis.db ".schema" > schema_dump.sql.txt ``` -Note that the produced database's schema shows bare table definitions; the meaning of the columns is -documented in the [Analyzer database reference](analyzer.md), which is also useful context. +The dump carries short notes on the non-obvious columns, because they are stored inside the `CREATE` +statements. That is usually enough to write correct queries. Attach the +[Analyzer database schema](analyzer-schema.md) as well when you want the full column reference. ## Related documentation | Topic | Description | |-------|-------------| | [Command-line tool](unitydatatool.md) | All commands and their options | -| [Analyzer database reference](analyzer.md) | Tables, views, and their columns | +| [Analyzer database schema](analyzer-schema.md) | Tables, views, and their columns | | [Example usage of Analyze](analyze-examples.md) | More worked queries | | [Comparing builds](comparing-builds.md) | Finding what changed between two builds | | [Overview of Unity Content](unity-content-format.md) | SerializedFiles, Archives, and TypeTrees | diff --git a/Documentation/analyze-examples-contentlayout.md b/Documentation/analyze-examples-contentlayout.md index 91ff1a4..07e2c75 100644 --- a/Documentation/analyze-examples-contentlayout.md +++ b/Documentation/analyze-examples-contentlayout.md @@ -81,7 +81,7 @@ WHERE filename = 'c0152db4dd710be51b2decb997325f34.cf'; ## Checking reference resolution (layout + content) When the layout was part of the analyze, the references between Content Files are resolved, so the -only expected entries in [`dangling_refs`](analyzer.md#dangling_refs--dangling_refs_view) are the +only expected entries in [`dangling_refs`](analyzer-schema.md#dangling_refs) are the references into Unity's built-in resources: ```sql diff --git a/Documentation/analyze-examples.md b/Documentation/analyze-examples.md index e4565f2..3821d2c 100644 --- a/Documentation/analyze-examples.md +++ b/Documentation/analyze-examples.md @@ -2,9 +2,9 @@ This topic gives some examples of using the SQLite output of the UnityDataTools Analyze command. -The command line arguments to invoke Analyze are documented [here](unitydatatool.md#analyzeanalyse). +The command line arguments to invoke Analyze are documented [here](command-analyze.md). -The definition of the views, and some internal details about how Analyze is implemented, can be found [here](analyzer.md). +The definition of the views is in the [Analyzer Database Schema](analyzer-schema.md); internal details about how Analyze is implemented are in the [Analyzer documentation](analyzer.md). Queries specific to ContentDirectory builds and `ContentLayout.json` are collected on a dedicated page: [Example queries for ContentDirectory builds](analyze-examples-contentlayout.md). @@ -18,7 +18,7 @@ However often it is useful to run queries from the command line, and to incorpor These examples assume you have `sqlite3` available in the path for your command prompt or terminal. On Windows that means that a directory containing `sqlite3.exe` is included in your PATH environmental variable. -On Windows, sqlite3.exe is available as part of the "SQLite command line tools", published from [www.sqlite.org](www.sqlite.org). +On Windows, sqlite3.exe is available as part of the "SQLite command line tools", published from [www.sqlite.org](https://www.sqlite.org). Note: Many of the examples in this topic assume that your database file is named `Analysis.db` in your current working directory. diff --git a/Documentation/analyzer-schema.md b/Documentation/analyzer-schema.md new file mode 100644 index 0000000..b05f37c --- /dev/null +++ b/Documentation/analyzer-schema.md @@ -0,0 +1,560 @@ +# Analyzer Database Schema + +Reference for the SQLite database produced by the [`analyze`](command-analyze.md) command. See +[Analyzer](analyzer.md) for an overview of the library, and [Example usage of +Analyze](analyze-examples.md) for worked queries. + +Parts of the schema are documented on their own pages: + +| Tables | Page | +|---|---| +| `content_layout*` | [ContentLayout in the Analyze Database](contentlayout-database.md) | +| `build_report*` | [BuildReport](buildreport.md) | +| `addressables_build*` | [Addressables Build Report Analysis](addressables-build-reports.md) | + +## How to read this page + +* **Start from the views.** They join the tables into the presentations you normally want, and a + GUI tool such as [DB Browser for SQLite](https://sqlitebrowser.org/) can display them directly. + `object_view` is the main entry point. +* **The database documents itself too.** Short notes for the non-obvious columns are stored inside + the `CREATE` statements, so `sqlite3 Analysis.db ".schema objects"` shows them. This page is the + full reference; the in-database notes are the reminder you get when you only have the `.db`. +* **Two columns hold `''` rather than NULL** when they have no value: `serialized_files.archive` + and `objects.game_object`. `object_view` maps the former to NULL. Test for `''` when querying the + tables directly. +* **Ids are analyzer-assigned.** `objects.id` and the ids that reference it exist only in the + database. The Unity object id is `objects.object_id`. +* **Two options change what gets populated.** `--skip-references` leaves `refs`, `dangling_refs` and + `script_object_view` empty. `--skip-crc` sets every `objects.crc32` to 0, which makes + `view_potential_duplicates` report many false positives. + +## At a glance + +Core tables: + +| Name | One row per | +|---|---| +| [`objects`](#objects) | Unity object found in the analyzed files | +| [`types`](#types) | distinct Unity type name | +| [`serialized_files`](#serialized_files) | SerializedFile analyzed | +| [`archives`](#archives) | Unity Archive analyzed | +| [`refs`](#refs) | reference from one object to another | +| [`dangling_refs`](#dangling_refs) | reference target that was never analyzed | +| [`property_names`](#property_names-and-property_types) | distinct property path used by `refs` | +| [`property_types`](#property_names-and-property_types) | distinct referenced type name used by `refs` | + +Core views: + +| Name | Purpose | +|---|---| +| [`object_view`](#object_view) | every object with its type, file and archive names resolved | +| [`refs_view`](#refs_view) | `refs` with the property strings resolved | +| [`dangling_refs_view`](#dangling_refs_view) | each unresolved target and the objects referencing it | +| [`view_breakdown_by_type`](#view_breakdown_by_type) | object count and total size per type | +| [`view_potential_duplicates`](#view_potential_duplicates) | objects that appear in more than one file | +| [`view_material_shader_refs`](#view_material_shader_refs-and-view_material_texture_refs) | each Material and its Shader | +| [`view_material_texture_refs`](#view_material_shader_refs-and-view_material_texture_refs) | each Material and its Textures | + +AssetBundle and MonoScript: + +| Name | Purpose | +|---|---| +| [`assetbundle_assets`](#assetbundle_assets) | assets explicitly assigned to an AssetBundle | +| [`assetbundle_asset_view`](#assetbundle_asset_view) | those assets with their object columns resolved | +| [`preload_dependencies`](#preload_dependencies) | objects Unity preloads alongside another object | +| [`preload_dependencies_view`](#preload_dependencies_view) | preload pairs with both sides resolved | +| [`monoscripts`](#monoscripts) | C# class behind each MonoBehaviour / ScriptableObject | +| [`monoscripts_view`](#monoscripts_view) | MonoScripts with their containing file | +| [`script_object_view`](#script_object_view) | MonoBehaviours and ScriptableObjects with their C# type | + +Type-specific views, each `object_view` plus extra columns: + +| Name | Type | +|---|---| +| [`animation_view`](#animation_view) | AnimationClip | +| [`audio_clip_view`](#audio_clip_view) | AudioClip | +| [`mesh_view`](#mesh_view) | Mesh | +| [`texture_view`](#texture_view) | Texture2D | +| [`shader_view`](#shader_view) | Shader | +| [`shader_subprogram_view`](#shader_subprogram_view) | Shader sub-programs | +| [`shader_keyword_ratios`](#shader_keyword_ratios) | Shader keyword impact on variant count | +| [`view_breakdowns_shaders`](#view_breakdowns_shaders) | Shaders aggregated by name | + +--- + +# Core tables + +## objects + +One row per Unity object discovered during analysis, holding the properties common to all types. +Query [`object_view`](#object_view) instead unless you need the raw ids. + +| Column | Type | Description | +|---|---|---| +| `id` | INTEGER | Analyzer-assigned id, unique across the database. Does not exist in the serialized data. Primary key. | +| `object_id` | INTEGER | The Unity local file id, serialized as `m_PathID` in object references (PPtrs). Signed 64-bit. Unique within its SerializedFile but not across files. | +| `serialized_file` | INTEGER | [`serialized_files.id`](#serialized_files) of the file containing the object. | +| `type` | INTEGER | [`types.id`](#types). | +| `name` | TEXT | The `Object.name` property. Empty for the many object types that have no name. | +| `game_object` | INTEGER | `objects.id` of the GameObject that owns this object. Set only for components; `''` otherwise. | +| `size` | INTEGER | Size in bytes, including any external `.resS` / `.resource` stream data belonging to the object. | +| `crc32` | INTEGER | CRC of the object's serialized state, including its external stream data. Useful for detecting whether an object changed between builds. `0` for every object when `--skip-crc` was used. | + +## types + +The Unity type names referenced by [`objects.type`](#objects). + +| Column | Type | Description | +|---|---|---| +| `id` | INTEGER | The Unity class id. `-1` is a synthetic type added by analyze, see below. Primary key. | +| `name` | TEXT | The type name, e.g. `Texture2D`. | + +Type `-1` is `Scene`. A scene has no single Unity object that represents it, so analyze inserts a +synthetic `Scene` object for scene bundles in order to have something for the AssetBundle container +entry and the scene's preload dependencies to hang off. + +## serialized_files + +One row per SerializedFile encountered during analysis. + +| Column | Type | Description | +|---|---|---| +| `id` | INTEGER | Analyzer-assigned id. Primary key. | +| `archive` | INTEGER | [`archives.id`](#archives) of the Unity Archive containing this file, or `''` when the file is not inside an archive. [`object_view`](#object_view) maps `''` to NULL. | +| `name` | TEXT | The file's name on disk, see below. | + +The name is usually a technical, hash-based string rather than a readable path, because that is how +the file is named on disk: + +* **Regular AssetBundles**: `CAB-`. This is MD4, not Unity's + `Hash128` (spooky hash), and the optional AssetBundle-filename hash is not part of it. +* **Scene bundles** vary by build pipeline: `BuildPipeline.BuildAssetBundles` produces + `BuildPlayer-`; the Scriptable Build Pipeline and Addressables produce + `CAB-`; the Multi-Process Build Pipeline produces `CAB-`. +* **Player builds** name scenes `level0`, `level1`, ... in scene-list order. + +A file recorded because it holds a [dangling reference](#dangling_refs) target also appears here, +with `archive` NULL and no objects of its own. In that case the lowercased file name is all that is +known about it. + +## archives + +One row per Unity Archive encountered during analysis. AssetBundles are archives, and compressed +Player and ContentDirectory builds also keep their content in archives. A bare SerializedFile that +is not inside an archive has no row here. + +| Column | Type | Description | +|---|---|---| +| `id` | INTEGER | Analyzer-assigned id. Primary key. | +| `name` | TEXT | The archive's name on the file system. UNIQUE. | +| `file_size` | INTEGER | Size of the archive file in bytes. | + +`name` is UNIQUE and case-sensitive because analyze supports a single build at a time: two archives +with the same name would make every query ambiguous. A duplicate is detected while writing and +reported as an error; the constraint is the durable backstop. See +[Comparing Builds](comparing-builds.md) for how to analyze several builds. + +## refs + +Every reference between Unity objects (PPtrs), whether the two objects are in the same SerializedFile +or in different ones. On large builds this is by far the biggest table, so the property strings are +deduplicated into [`property_names`](#property_names-and-property_types) and +[`property_types`](#property_names-and-property_types) and only integer ids are stored here. Query +[`refs_view`](#refs_view) to get the strings back. + +| Column | Type | Description | +|---|---|---| +| `object` | INTEGER | [`objects.id`](#objects) of the referencing object. | +| `referenced_object` | INTEGER | [`objects.id`](#objects) of the target, or [`dangling_refs.id`](#dangling_refs) when the target's file was not analyzed. | +| `property_path` | INTEGER | [`property_names.id`](#property_names-and-property_types). | +| `property_type` | INTEGER | [`property_types.id`](#property_names-and-property_types). | + +Not populated when analyze is run with `--skip-references`. + +## dangling_refs + +When a reference points at an object whose SerializedFile was not part of the analyzed input, analyze +still assigns that target an object id but never writes an [`objects`](#objects) row for it. +`dangling_refs` records those targets, so the reference information is preserved and every object id +is accounted for: each id resolves to exactly one of `objects` or `dangling_refs`, never both. + +| Column | Type | Description | +|---|---|---| +| `id` | INTEGER | The assigned object id. No `objects` row has this id. Primary key. | +| `object_id` | INTEGER | The target's local file id (`m_PathID`) within its SerializedFile. | +| `serialized_file` | INTEGER | [`serialized_files.id`](#serialized_files) of the un-analyzed file. That row has `archive` NULL and no objects of its own. | + +Common causes: + +* Analyzing a single AssetBundle or a partial subset of a bundle group, so cross-bundle references + point at bundles that were not analyzed. +* A Player build referencing `unity default resources` - a built-in file that ships without + TypeTrees and so cannot be analyzed. (`Resources/unity_builtin_extra` is built alongside your + content and can be analyzed, but produces the same dangling references when it is not part of the + analyzed set.) +* A ContentDirectory build analyzed without its `ContentLayout.json`, so the symbolic external + references in the content files cannot be resolved. See + [ContentLayout in the Analyze Database](contentlayout-database.md). + +Not populated when analyze is run with `--skip-references`. + +## property_names and property_types + +Deduplicated lookup tables for the strings that [`refs`](#refs) would otherwise repeat on every row. +Both have the same shape: + +| Column | Type | Description | +|---|---|---| +| `id` | INTEGER | Primary key, referenced by `refs`. | +| `name` | TEXT | The string. `property_names` holds property paths (e.g. `m_Shader`, `m_Materials[0]`); `property_types` holds referenced type names (e.g. `Texture2D`, `MonoScript`). | + +Prefer [`refs_view`](#refs_view) over joining these by hand. + +--- + +# Core views + +## object_view + +The main view: every object in the analyzed build output, with its type, SerializedFile and archive +names resolved. The output could be AssetBundles, a ContentDirectory build, a Player build, or +standalone SerializedFiles. + +| Column | Type | Description | +|---|---|---| +| `id` | INTEGER | [`objects.id`](#objects). | +| `object_id` | INTEGER | The Unity object id. | +| `archive` | TEXT | Name of the Unity Archive containing the object, or NULL when the source file was a bare SerializedFile. | +| `serialized_file` | TEXT | Name of the SerializedFile containing the object. | +| `type` | TEXT | The object's type name. | +| `name` | TEXT | The object's name, if it had one. | +| `game_object` | INTEGER | `objects.id` of the containing GameObject, if any. Mostly for components. | +| `size` | INTEGER | Size in bytes, e.g. `3343772`. | +| `pretty_size` | TEXT | Size in a readable form, e.g. `3.2 MB`. | +| `crc32` | INTEGER | The object's CRC. | + +## refs_view + +[`refs`](#refs) with the `property_path` and `property_type` ids resolved to their strings. Use this +rather than joining the lookup tables yourself. + +| Column | Type | Description | +|---|---|---| +| `object` | INTEGER | The referencing object's id. | +| `referenced_object` | INTEGER | The target's id. | +| `property_path` | TEXT | The property holding the reference, e.g. `m_Shader`. | +| `property_type` | TEXT | The referenced type, e.g. `Texture2D`. | + +```sql +SELECT * FROM refs_view WHERE property_type = 'MonoScript'; +``` + +## dangling_refs_view + +Each [dangling target](#dangling_refs) joined back to the object or objects that reference it: one +row per (referencing object, dangling target) reference. + +| Column | Type | Description | +|---|---|---| +| `source_id` | INTEGER | `objects.id` of the referencing object. | +| `source_serialized_file` | TEXT | The referencing object's file. | +| `source_object_id` | INTEGER | The referencing object's Unity object id. | +| `property_path` | TEXT | The property holding the reference. | +| `property_type` | TEXT | The referenced type. | +| `target_id` | INTEGER | `dangling_refs.id` of the unresolved target. | +| `target_serialized_file` | TEXT | Name of the file the target lives in. | +| `target_object_id` | INTEGER | The target's local file id. | + +```sql +-- what does object 42 fail to resolve, and where should those objects have come from? +SELECT * FROM dangling_refs_view WHERE source_id = 42; +``` + +Because the view joins `refs`, it is empty when analyze is run with `--skip-references` (in that mode +neither `refs` nor `dangling_refs` are populated). + +## view_breakdown_by_type + +Total number and size of the objects, aggregated by type, largest first. + +| Column | Type | Description | +|---|---|---| +| `type` | TEXT | The type name. | +| `count` | INTEGER | Number of objects of this type. | +| `byte_size` | INTEGER | Total size in bytes. | +| `pretty_size` | TEXT | Total size in a readable form. | + +## view_potential_duplicates + +Objects that are possibly included more than once in the build output. This happens when an asset is +referenced from multiple AssetBundles but is not assigned to one: Unity then copies it into every +bundle that references it. + +| Column | Type | Description | +|---|---|---| +| `instances` | INTEGER | How many copies were found. | +| `name` | TEXT | The object's name. | +| `type` | TEXT | The object's type. | +| `pretty_total_size` | TEXT | Combined size of all copies, readable form. | +| `total_size` | INTEGER | Combined size of all copies in bytes. | +| `size` | INTEGER | Size of one copy in bytes. | +| `pretty_size` | TEXT | Size of one copy, readable form. | +| `in_files` | TEXT | The archives (or SerializedFiles) the copies were found in. | + +Rows are grouped by name, type, size *and* `crc32`, so the view is normally very accurate. With +`--skip-crc` every `crc32` is 0 and the view reports many false positives. + +## view_material_shader_refs and view_material_texture_refs + +Each Material paired with the Shader it references, and with the Textures it references. Both +include the Material's AssetBundle container path when it has one. + +`view_material_shader_refs`: `material_id`, `material_name`, `material_path`, `material_archive`, +`shader_id`, `shader_name`, `shader_archive`. + +`view_material_texture_refs`: `material_id`, `material_name`, `material_path`, `material_archive`, +`texture_id`, `texture_name`, `texture_archive`. + +--- + +# AssetBundle tables and views + +## assetbundle_assets + +The assets an AssetBundle explicitly exposes: one row per entry in the AssetBundle object's +`m_Container`. Dependencies that Unity pulled in automatically at build time are *not* listed here, +see [`preload_dependencies`](#preload_dependencies). + +| Column | Type | Description | +|---|---|---| +| `object` | INTEGER | [`objects.id`](#objects) of the asset. For a scene bundle this is the synthetic [`Scene`](#types) object. | +| `name` | TEXT | The container path the asset is addressed by. | + +This data comes from the AssetBundle Unity object, so the table is populated only for AssetBundle +builds. Player and ContentDirectory builds have no such object and it is empty for them. + +For a scene bundle the container entry names the scene (its `.unity` path) and points at the +synthetic `Scene` object. This applies to both `BuildPipeline.BuildAssetBundles` and Scriptable +Build Pipeline / Addressables scene bundles. + +## assetbundle_asset_view + +[`assetbundle_assets`](#assetbundle_assets) joined to [`object_view`](#object_view): the columns of +`object_view` plus `asset_name`, the container path. + +The join is an INNER JOIN, so a container entry whose object is not in the `objects` table is +omitted. In practice that only happens when the object was genuinely not analyzed, for example it +lives in a bundle that was not part of the analyzed set. The underlying table still holds those rows. + +## preload_dependencies + +Preload relationships as `object` to `dependency` pairs. A "dependency" here is an object that Unity +preloads or pulls in alongside another object. + +| Column | Type | Description | +|---|---|---| +| `object` | INTEGER | [`objects.id`](#objects). What this is depends on the build type, see below. | +| `dependency` | INTEGER | [`objects.id`](#objects) of the preloaded object, or [`dangling_refs.id`](#dangling_refs) when it was not analyzed. | + +The rows come from the AssetBundle and PreloadData Unity objects, so the table is populated for +AssetBundle **and Player** builds, but not ContentDirectory builds, which have neither object. What +the `object` side is depends on the build: + +* **AssetBundle asset**: the explicitly-assigned asset, with its dependencies taken from the + AssetBundle object's `m_PreloadTable`. +* **Scene bundle**: the synthetic [`Scene`](#types) object. Its dependencies are the scene's shared + assets and the entries of the scene's PreloadData. This works for both + `BuildPipeline.BuildAssetBundles` and Scriptable Build Pipeline / Addressables scene bundles. The + scene's own content objects (GameObjects and so on) are not listed as dependencies, but they share + the scene object's `serialized_file`. +* **Player build**: the `PreloadData` object itself, because a Player build has no scene object to + attach the dependencies to. A Player build has one `PreloadData` per scene (in its + `sharedassetsN.assets`) plus one in `globalgamemanagers.assets` for the always-loaded set. + +The `dependency` side often references an object analyze never recorded, most commonly objects in +`unity default resources`. Those ids are catalogued in [`dangling_refs`](#dangling_refs). + +## preload_dependencies_view + +A convenience view over [`preload_dependencies`](#preload_dependencies) that resolves the ids: it +joins each row's `object` to [`assetbundle_asset_view`](#assetbundle_asset_view) and its `dependency` +to [`object_view`](#object_view). Columns: `id`, `asset_name`, `archive`, `type`, `dep_id`, +`dep_archive`, `dep_name`, `dep_type`. + +Filter by `id` or `asset_name` for the dependencies of one asset, or by `dep_id` for everything that +depends on a given object - useful for working out why an object was included in a build. + +Because of those inner joins the view is narrower than the underlying table: + +* It only includes rows whose `object` is an AssetBundle asset or scene. Player-build rows hang off a + `PreloadData` object rather than an AssetBundle asset, so they do not appear here - query the + `preload_dependencies` table directly for those. +* It drops rows whose `dependency` is a dangling id, because those have no `object_view` row to join + to. + +--- + +# MonoScript tables and views + +## monoscripts + +The class information for all the C# types of MonoBehaviour objects in the build output, including +ScriptableObjects: assembly name, C# namespace and class name. + +## monoscripts_view + +`monoscripts` joined so you can see which AssetBundle or SerializedFile contains each MonoScript +object. + +## script_object_view + +All the MonoBehaviour and ScriptableObject objects in the build output with their location, size and +precise C# type, built from the `monoscripts` and [`refs`](#refs) tables. Empty when analyze is run +with `--skip-references`. + +--- + +# Type-specific views + +Each of these has the same columns as [`object_view`](#object_view) plus the ones listed. + +## animation_view + +AnimationClips. + +| Column | Description | +|---|---| +| `legacy` | 1 if it is a legacy animation, 0 otherwise | +| `events` | the number of events | + +## audio_clip_view + +AudioClips. + +| Column | Description | +|---|---| +| `bits_per_sample` | number of bits per sample | +| `frequency` | sampling frequency | +| `channels` | number of channels | +| `load_type` | `Compressed in Memory`, `Decompress on Load` or `Streaming` | +| `format` | compression format | + +## mesh_view + +Meshes. + +| Column | Description | +|---|---| +| `sub_meshes` | the number of sub-meshes | +| `blend_shapes` | the number of blend shapes | +| `bones` | the number of bones | +| `indices` | the number of vertex indices | +| `vertices` | the number of vertices | +| `compression` | 1 if compressed, 0 otherwise | +| `rw_enabled` | 1 if the mesh has the *R/W Enabled* option, 0 otherwise | +| `vertex_size` | number of bytes used by each vertex | +| `channels` | name and type of the vertex channels | + +## texture_view + +Texture2Ds. + +| Column | Description | +|---|---| +| `width` / `height` | texture resolution | +| `format` | compression format | +| `mip_count` | number of mipmaps | +| `rw_enabled` | 1 if the texture has the *R/W Enabled* option, 0 otherwise | + +## shader_view + +Shaders. + +| Column | Description | +|---|---| +| `decompressed_size` | approximate size in bytes the shader needs at runtime when loaded | +| `sub_shaders` | the number of sub-shaders | +| `sub_programs` | the number of sub-programs (usually one per shader variant, stage and pass) | +| `unique_programs` | the number of unique programs (variants with identical programs share one program in memory) | +| `keywords` | list of all the keywords affecting the shader | + +## shader_subprogram_view + +One row per shader sub-program. Same columns as [`shader_view`](#shader_view) plus: + +| Column | Description | +|---|---| +| `api` | the graphics API, e.g. DX11, Metal, GLES | +| `pass` | the pass number of the sub-program | +| `pass_name` | the pass name, if available | +| `hw_tier` | the hardware tier (as defined in the Graphics settings) | +| `shader_type` | the type of shader, e.g. vertex, fragment | +| `sub_program` | the sub-program index for this pass and shader type | +| `keywords` | the shader keywords specific to this sub-program | + +## shader_keyword_ratios + +Helps determine which shader keywords are causing a large number of variants. Define a "program" as a +unique combination of shader, sub-shader, hardware tier, pass number, API and shader type. Each row +of this view is one program paired with one of its keywords. + +| Column | Description | +|---|---| +| `shader_id` | the shader id | +| `name` | the shader name | +| `sub_shader` | the sub-shader number | +| `hw_tier` | the hardware tier | +| `pass` | the pass number | +| `api` | the graphics API | +| `pass_name` | the pass name, if available | +| `shader_type` | the type of shader | +| `total_variants` | total number of variants for this program | +| `keyword` | one of the program's keywords | +| `variants` | number of variants including this keyword | +| `ratio` | `variants` / `total_variants` | + +Use `ratio` to judge how a keyword affects the variant count. At 0.5 the keyword is in half the +variants, which means it is not being stripped at all: every variant exists both with and without +it. Keywords with a ratio close to 0.5 are therefore the best stripping targets. A ratio close to 0 +or 1 means the keyword is in almost none or almost all of the variants, and stripping it will not +make much difference. + +## view_breakdowns_shaders + +All the shaders aggregated by name. `instances` indicates how many times the shader was found in the +data files. Also provides the total size per shader and the list of AssetBundles it was found in. + +--- + +# Schema version + +The database records which schema it was produced with in `PRAGMA user_version`. Commands that read +an existing database (currently only [`find-refs`](command-find-refs.md)) compare against it to give +a clean error instead of failing on a missing table or column. + +Any schema change - a new or changed table, view or column - must bump the pragma in +`Analyzer/Resources/Init.sql` and add a row here. Databases produced before versioning report 0. + +| Version | Change | +|---|---| +| 1 | Normalized `refs` table ([#44](https://github.com/Unity-Technologies/UnityDataTools/issues/44)) | +| 2 | Renamed `assets` / `asset_dependencies` to `assetbundle_assets` / `preload_dependencies` ([#82](https://github.com/Unity-Technologies/UnityDataTools/issues/82)) | +| 3 | Renamed the `asset_bundles` table to `archives`, and the `asset_bundle` column/alias to `archive` ([#68](https://github.com/Unity-Technologies/UnityDataTools/issues/68)) | +| 4 | `build_report_packed_asset_contents_view` type column changed from numeric id to type name ([#55](https://github.com/Unity-Technologies/UnityDataTools/issues/55)) | +| 5 | Added the `dangling_refs` table and view ([#85](https://github.com/Unity-Technologies/UnityDataTools/issues/85)) | +| 6 | `archives.name` is UNIQUE ([#51](https://github.com/Unity-Technologies/UnityDataTools/issues/51)) | +| 7 | Unity 6.6 `build_reports` columns and `build_report_content_*` tables ([#107](https://github.com/Unity-Technologies/UnityDataTools/issues/107)); `asset_name` / `asset_extension` columns on `build_report_source_assets` ([#110](https://github.com/Unity-Technologies/UnityDataTools/issues/110)) | + +## Related documentation + +| Topic | Page | +|---|---| +| [Analyzer](analyzer.md) | The library, and how to extend it | +| [analyze command](command-analyze.md) | Running `analyze` and its options | +| [Example usage of Analyze](analyze-examples.md) | Worked queries | +| [ContentLayout in the Analyze Database](contentlayout-database.md) | The `content_layout*` tables | +| [BuildReport](buildreport.md) | The `build_report*` tables | +| [Addressables Build Report Analysis](addressables-build-reports.md) | The `addressables_build*` tables | +| [Comparing Builds](comparing-builds.md) | Analyzing more than one build | +| [Using UnityDataTool with an AI Agent](agent-guide.md) | Agent-oriented workflow | diff --git a/Documentation/analyzer.md b/Documentation/analyzer.md index 099a2c6..fdf5154 100644 --- a/Documentation/analyzer.md +++ b/Documentation/analyzer.md @@ -4,7 +4,7 @@ The Analyzer is a class library that can be used to analyze the content of Unity as AssetBundles and SerializedFiles. It iterates through all the serialized objects and uses the TypeTree to extract information about these objects (e.g. name, size, etc.) -The most common use of this library is through the [analyze](unitydatatool.md#analyzeanalyse) +The most common use of this library is through the [analyze](command-analyze.md) command of the UnityDataTool. This uses the Analyze library to generate a SQLite database. Once generated, a tool such as the [DB Browser for SQLite](https://sqlitebrowser.org/), or the command line `sqlite3` tool, can be used to look at the content of the database. @@ -13,296 +13,32 @@ Once generated, a tool such as the [DB Browser for SQLite](https://sqlitebrowser See [this topic](analyze-examples.md) for examples of how to use the SQLite output of the UnityDataTool Analyze command. -# Database Schema Reference +# Database Schema -The database provides different tables and views. The views join multiple tables together to provide useful presentations of the data. With GUI-based SQLite tools it is often possible to skip writing your own SQL queries, and simply view the content of these pre-defined queries. +The database provides tables and views. The views join multiple tables together to provide useful +presentations of the data, and with a GUI-based SQLite tool it is often possible to skip writing +queries altogether and simply browse them. `object_view` is the main entry point: one row per +analyzed object, with its type, SerializedFile and archive names already resolved. -This section gives an overview of the main tables and views. +The schema is documented in full on these pages: -## object_view +| Tables and views | Page | +|---|---| +| Core schema: objects, types, files, archives, references, AssetBundle and type-specific views | [Analyzer Database Schema](analyzer-schema.md) | +| `content_layout*` (ContentDirectory builds) | [ContentLayout in the Analyze Database](contentlayout-database.md) | +| `build_report*` | [BuildReport](buildreport.md) | +| `addressables_build*` | [Addressables Build Report Analysis](addressables-build-reports.md) | -This is the main view, providing information about all the objects in the analyzed build output. -That output could be AssetBundles, a ContentDirectory build, a Player build, or standalone -SerializedFiles. -Its columns are: -* id: a unique id without any meaning outside of the database -* object_id: the Unity object id (unique inside its SerializedFile but not necessarily across all - the files in the build) -* archive: the name of the Unity Archive containing the object (will be null if the source file - was a bare SerializedFile and not inside an archive). Archives back AssetBundles, ContentDirectory - builds and Player builds. -* serialized_file: the name of the SerializedFile containing the object -* type: the type of the object -* name: the name of the object, if it had one -* game_object: the id of the GameObject containing this object, if any (mostly for Components) -* size: the size of the object in bytes (e.g. 3343772) -* pretty_size: the size in an easier to read format (e.g. 3.2 MB) - -## archives - -One row per Unity Archive encountered during analysis, holding the archive file's `name` and -`file_size`. AssetBundle files are recorded here. Compressed Player and other builds may also have content inside archive files. Each SerializedFile links to its archive through the `serialized_files.archive` -column, and `object_view` surfaces the archive name as its `archive` column. A bare SerializedFile that is not inside an archive has no row here. - -## view_breakdown_by_type - -This view lists the total number and size of the objects, aggregated by type. - -## view_potential_duplicates - -This view lists the objects that are possibly included more than once in the AssetBundles. This can -happen when an asset is referenced from multiple AssetBundles but is not assigned to one. In this -case, Unity will include the asset in all the AssetBundles with a reference to it. The -view_potential_duplicates provides the number of instances and the total size of the potentially -duplicated assets. It also lists all the AssetBundles where the asset was found. - -If the `--skip-crc` option is used, there will be a lot of false positives in that view. Otherwise, -it should be very accurate because CRCs are used to determine if objects are identical. - -## assetbundle_asset_view (AssetBundleProcessor) - -Lists the assets that were explicitly assigned to AssetBundles, one row per entry in the AssetBundle -object's `m_Container`. Dependencies that Unity pulled in automatically at build time are not listed -here (see `preload_dependencies_view`). The columns are those of *object_view* plus `asset_name`, -the container path of the asset. - -This data comes from the AssetBundle Unity object, so the view is only populated for AssetBundle -builds; Player and ContentDirectory builds have no such object and it is empty for them. For a scene -bundle the container entry names the scene (its `.unity` path) and points at a synthetic object of -type `Scene` (there is no single Unity object that represents a scene). Scene objects are created -for both BuildPipeline.BuildAssetBundles and Scriptable Build Pipeline / Addressables scene bundles. - -The view is built with an INNER JOIN to *object_view*, so a container entry whose object is not in -the `objects` table is omitted. In practice that only happens if the object was genuinely not -analyzed (for example it lives in a bundle that was not part of the analyzed set); the underlying -`assetbundle_assets` table still holds those rows. - -## preload_dependencies - -This table records preload relationships as `object` -> `dependency` id pairs, where both are -`objects.id` values. A "dependency" is an object that Unity preloads / pulls in alongside another -object. The rows come from the AssetBundle and PreloadData Unity objects, so the table is populated -for AssetBundle and Player builds but not ContentDirectory builds (which have neither object). - -What the `object` is depends on the build: - -- **AssetBundle asset**: the explicitly-assigned asset, with its dependencies taken from the - AssetBundle object's preload table. -- **Scene bundle**: a synthetic `Scene` object (a scene has no single Unity object). Its - dependencies are the scene's shared assets and the entries of the scene's PreloadData. This works - for both BuildPipeline.BuildAssetBundles and Scriptable Build Pipeline / Addressables scene - bundles. The scene's own content objects (GameObjects, etc.) are not listed as dependencies but - share the scene object's `serialized_file`. -- **Player build**: the `PreloadData` object itself, because a player build has no scene object to - attach the dependencies to. A player build has one `PreloadData` per scene (in its - `sharedassetsN.assets`) plus one in `globalgamemanagers.assets` for the always-loaded set. - -The `dependency` side can reference an object that analyze never recorded, leaving a "dangling" id -with no matching `objects` row (such ids are catalogued in the `dangling_refs` table). The most -common case is objects in `unity default resources`, the -built-in resource file that ships with the Unity Editor without TypeTrees and so cannot be analyzed -without a specially built copy. (`Resources/unity_builtin_extra` is built alongside your content and -can be analyzed, but produces the same dangling references when it is not part of the analyzed set.) - -## preload_dependencies_view (AssetBundleProcessor) - -A convenience view over `preload_dependencies` that resolves the ids to readable columns: it joins -each row's `object` to `assetbundle_asset_view` and its `dependency` to *object_view*. Filter by `id` -or `asset_name` for the dependencies of one asset, or by `dep_id` for everything that depends on a -given object (useful for figuring out why an object was included in a build). - -Because of those inner joins the view is narrower than the underlying table: - -- It only includes rows whose `object` is an AssetBundle asset or scene (i.e. present in - `assetbundle_asset_view`). Player-build rows hang off a `PreloadData` object rather than an - AssetBundle asset, so they do not appear here - query the `preload_dependencies` table directly. -- It drops rows whose `dependency` is a dangling id (see above), because those have no *object_view* - row to join to. - -## monoscripts - -Show the class information for all the C# types of MonoBehaviour objects in the build output (including ScriptableObjects). - -This includes the assembly name, C# namespace and class name. - -## monoscripts_view - -This view is a convenient view for seeing which AssetBundle / SerializedFile contains each MonoScript object. - -## script_object_view - -This view lists all the MonoBehaviour and ScriptableObject objects in the build output, with their location, size and precise C# type (using the `monoscripts` and `refs` tables). This view is not populated if analyze is run with the `--skip-references` option. - -## animation_view (AnimationClipProcessor) - -This provides additional information about AnimationClips. The columns are the same as those in -the *object_view*, with the addition of: -* legacy: 1 if it's a legacy animation, 0 otherwise -* events: the number of events - -## audio_clip_view (AudioClipProcessor) - -This provides additional information about AudioClips. The columns are the same as those in -the *object_view*, with the addition of: -* bits_per_sample: number of bits per sample -* frequency: sampling frequency -* channels: number of channels -* load_type: either *Compressed in Memory*, *Decompress on Load* or *Streaming* -* format: compression format - -## mesh_view (MeshProcessor) - -This provides additional information about Meshes. The columns are the same as those in -the *object_view*, with the addition of: -* sub_meshes: the number of sub-meshes -* blend_shapes: the number of blend shapes -* bones: the number of bones -* indices: the number of vertex indices -* vertices: the number of vertices -* compression: 1 if compressed, 0 otherwise -* rw_enabled: 1 if the mesh has the *R/W Enabled* option, 0 otherwise -* vertex_size: number of bytes used by each vertex -* channels: name and type of the vertex channels - -## texture_view (Texture2DProcessor) - -This provides additional information about Texture2Ds. The columns are the same as those in -the *object_view*, with the addition of: -* width/height: texture resolution -* format: compression format -* mip_count: number of mipmaps -* rw_enabled: 1 if the mesh has the *R/W Enabled* option, 0 otherwise - -## shader_view (ShaderProcessor) - -This provides additional information about Shaders. The columns are the same as those in -the *object_view*, with the addition of: -* decompressed_size: the approximate size in bytes that this shader will need at runtime when - loaded -* sub_shaders: the number of sub-shaders -* sub_programs: the number of sub-programs (usually one per shader variant, stage and pass) -* unique_programs: the number of unique program (variants with identical programs will share the - same program in memory) -* keywords: list of all the keywords affecting the shader - -## shader_subprogram_view (ShaderProcessor) - -This view lists all the shader sub-programs and has the same columns as the *shader_view* with the -addition of: -* api: the API of the shader (e.g. DX11, Metal, GLES, etc.) -* pass: the pass number of the sub-program -* pass_name: the pass name, if available -* hw_tier: the hardware tier of the sub-program (as defined in the Graphics settings) -* shader_type: the type of shader (e.g. vertex, fragment, etc.) -* sub_program: the subprogram index for this pass and shader type -* keywords: the shader keywords specific to this sub-program - -## shader_keyword_ratios - -This view can help to determine which shader keywords are causing a large number of variants. To -understand how it works, let's define a "program" as a unique combination of shader, subshader, -hardware tier, pass number, API (DX, Metal, etc.), and shader type (vertex, fragment, etc). - -Each row of the view corresponds to a combination of one program and one of its keywords. The -columns are: - -* shader_id: the shader id -* name: the shader name -* sub_shader: the sub-shader number -* hw_tier: the hardware tier of the sub-program (as defined in the Graphics settings) -* pass: the pass number of the sub-program -* api: the API of the shader (e.g. DX11, Metal, GLES, etc.) -* pass_name: the pass name, if available -* shader_type: the type of shader (e.g. vertex, fragment, etc.) -* total_variants: total number of variants for this program. -* keyword: one of the program's keywords -* variants: number of variants including this keyword. -* ratio: variants/total_variants - -The ratio can be used to determine how a keyword affects the number of variants. When it is equal -to 0.5, it means that it is in half of the variants. Basically, that means that it is not stripped -at all because each of the program's variants has a version with and without that keyword. -Therefore, keywords with a ratio close to 0.5 are good targets for stripping. When the ratio is -close to 0 or 1, it means that the keyword is in almost none or almost all of the variants and -stripping it won't make a big difference. - -## view_breakdowns_shaders (ShaderProcessor) - -This view lists all the shaders aggregated by name. The *instances* column indicates how many time -the shader was found in the data files. It also provides the total size per shader and the list of -AssetBundles in which they were found. - -## refs / refs_view - -The `refs` table records the references between objects: for each reference it stores the source -`object`, the `referenced_object`, and the property that holds the reference. On large builds this -table dominates the database size, so the property strings are deduplicated into two lookup tables -and `refs` stores integer ids into them: - -* `property_names`: distinct property paths (e.g. `m_Shader`, `m_Materials[0]`). -* `property_types`: distinct referenced types (e.g. `Texture2D`, `MonoScript`). - -The `refs_view` rejoins these so the original strings are available directly. Query `refs_view` -(columns `object`, `referenced_object`, `property_path`, `property_type`) rather than joining the -lookup tables by hand: - -```sql -SELECT * FROM refs_view WHERE property_type = 'MonoScript'; -``` - -These tables are not populated when analyze is run with `--skip-references`. - -## dangling_refs / dangling_refs_view - -When a reference points to an object whose serialized file was not part of the analyzed input, -analyze still assigns that target an object id but never writes an `objects` row for it. `dangling_refs` -records those targets so the reference information is preserved and every object id is accounted for -(each id resolves to exactly one of `objects` or `dangling_refs`, never both). Common causes: - -* Analyzing a single AssetBundle or a partial subset of a bundle group, so cross-bundle references - point at bundles that were not analyzed. -* A Player build referencing `unity default resources` (and `Resources/unity_builtin_extra` when it - is not part of the analyzed set) - built-in files that ship without TypeTrees and are not analyzed. -* A ContentDirectory build analyzed without its `ContentLayout.json`, so references cannot be resolved. - -The columns mirror the `objects` table: `id` (the assigned object id, absent from `objects`), -`object_id` (the target's local file id / LFID) and `serialized_file`. The target file is recorded in -`serialized_files` (with `archive` NULL and no objects of its own) so its name is available; the -lowercased file name is all that is known about it - there is no type, size, or other detail. - -`dangling_refs_view` joins each dangling target back to the object(s) that reference it, one row per -reference, with columns `source_id`, `source_serialized_file`, `source_object_id`, `property_path`, -`property_type`, `target_id`, `target_serialized_file`, `target_object_id`. - -```sql --- what does object 42 fail to resolve, and where should those objects have come from? -SELECT * FROM dangling_refs_view WHERE source_id = 42; -``` - -Because the view joins `refs`, it is not populated when analyze is run with `--skip-references` -(in that mode neither `refs` nor `dangling_refs` are populated). - -## ContentLayout (content_layout tables) - -When the analyzed input includes the `ContentLayout.json` of a ContentDirectory build (see -[contentlayout.md](contentlayout.md)), its content is imported into a set of `content_layout*` -tables and views: the source assets each file was built from, the dependencies between the files, -the loadable objects and scenes, and the size of every artifact, all connected to the analyzed -objects in the core tables. These tables are only created when a layout is actually imported. - -See [ContentLayout in the Analyze Database](contentlayout-database.md) for the tables, the views, -and how they join with the core tables. - -## BuildReport - -See [BuildReport.md](buildreport.md) for details of the tables and views related to analyzing BuildReport files. +A produced database also carries short notes on its non-obvious columns inside the `CREATE` +statements, so `sqlite3 Analysis.db ".schema objects"` describes the columns without needing these +pages. The pages remain the full reference. # Advanced ## Using the library The [AnalyzerTool](../Analyzer/AnalyzerTool.cs) class is the API entry point. The main method is called -Analyze. It is currently hard coded to write using the [SQLiteWriter](../Analyzer/SQLite/SQLiteWriter.cs), +Analyze. It is currently hard coded to write using the [SQLiteWriter](../Analyzer/SQLite/Writers/SQLiteWriter.cs), but this approach could be extended to add support for other outputs. Calling this method processes the provided paths, which can be individual files or directories. @@ -312,9 +48,8 @@ basic information such as the size and the name of the object (if it has one). ## Extending the Library -The extracted information is forwarded to an object implementing the [IWriter](../Analyzer/IWriter.cs) -interface. The library provides the [SQLiteWriter](../Analyzer/SQLite/SQLiteWriter.cs) implementation that -writes the data into a SQLite database. +The extracted information is forwarded to the [SQLiteWriter](../Analyzer/SQLite/Writers/SQLiteWriter.cs), +which writes it into a SQLite database. The core properties that apply to all Unity Objects are extracted into the `objects` table. However much of the most useful Analyze functionality comes by virtue of the type-specific information that is extracted for @@ -322,18 +57,18 @@ important types like Meshes, Shaders, Texture2D and AnimationClips. For example File, then rows are added to both the `objects` table and the `meshes` table. The meshes table contains columns that only apply to Mesh objects, for example the number of vertices, indices, bones, and channels. The `mesh_view` is a view that joins the `objects` table with the `meshes` table, so that you can see all the properties of a Mesh object in one place. Each supported Unity object type follows the same pattern: -* A Handler class in the SQLite/Handlers, e.g. [MeshHandler.cs](../Analyzer/SQLite/Handler/MeshHandler.cs). +* A Handler class in the SQLite/Handlers, e.g. [MeshHandler.cs](../Analyzer/SQLite/Handlers/MeshHandler.cs). * The registration of the handler in the m_Handlers dictionary in [SerializedFileSQLiteWriter.cs](../Analyzer/SQLite/Writers/SerializedFileSQLiteWriter.cs). -* SQL statements defining extra tables and views associated with the type, e.g. [Mesh.sql](../Analyzer/SQLite/Resources/Mesh.sql). +* SQL statements defining extra tables and views associated with the type, e.g. [Mesh.sql](../Analyzer/Resources/Mesh.sql). * A Reader class that uses RandomAccessReader to read properties from the serialized object, e.g. [Mesh.cs](../Analyzer/SerializedObjects/Mesh.cs). -It would be possible to extend the Analyze library to add additional columns for the existing types, or by following the same pattern to add additional types. The [dump](unitydatatool.md#dump) feature of UnityDataTool is a useful way to see the property names and other details of the serialization for a type. Based on that information, code in the Reader class can use the RandomAccessReader to retrieve those properties to bring them into the SQLite database. +It would be possible to extend the Analyze library to add additional columns for the existing types, or by following the same pattern to add additional types. The [dump](command-dump.md) feature of UnityDataTool is a useful way to see the property names and other details of the serialization for a type. Based on that information, code in the Reader class can use the RandomAccessReader to retrieve those properties to bring them into the SQLite database. ## Supporting Other File Formats Another direction of possible extension is to support analyzing additional file formats, beyond Unity SerializedFiles. -This the approach taken to analyze Addressables Build Layout files, which are JSON files using the format defined in [BuildLayout.cs](../Analyzer/SQLite/Parsers/Models/BuildLayout.cs). +This the approach taken to analyze Addressables Build Layout files, which are JSON files using the format defined in [BuildLayout.cs](../UnityDataModels/BuildLayout.cs). Support for another file format could be added by deriving an additional class from SQLiteWriter and implementing a class derived from ISQLiteFileParser. Then follow the existing code structure convention to add new Commands (derived from AbstractCommand) and Resource .sql files to establish additional tables in the database. diff --git a/Documentation/assetbundle-format.md b/Documentation/assetbundle-format.md index 7e03089..76b4bcd 100644 --- a/Documentation/assetbundle-format.md +++ b/Documentation/assetbundle-format.md @@ -325,7 +325,7 @@ large dependency graphs by loading some assets on demand (via Addressables, Asse ## How `analyze` represents this -The [`analyze`](analyzer.md) command turns the above into queryable tables: +The [`analyze`](command-analyze.md) command turns the above into queryable tables: - Each object is linked to both its `serialized_file` and, when applicable, its containing `archive` in `object_view`. @@ -337,4 +337,4 @@ The [`analyze`](analyzer.md) command turns the above into queryable tables: for it, so scenes appear in `assetbundle_asset_view` and their PreloadData dependencies appear in `preload_dependencies_view`. -See [Analyzer](analyzer.md) for the full schema and the exact behaviour of those views. +See the [Analyzer Database Schema](analyzer-schema.md) for the full schema and the exact behaviour of those views. diff --git a/Documentation/command-analyze.md b/Documentation/command-analyze.md index 6ddd803..4a26dc9 100644 --- a/Documentation/command-analyze.md +++ b/Documentation/command-analyze.md @@ -62,7 +62,7 @@ Fastest analysis (skip both reference extraction and CRC): UnityDataTool analyze /path/to/bundles --skip-references --skip-crc ``` -See also [Analyze Examples](../../Documentation/analyze-examples.md). +See also [Analyze Examples](analyze-examples.md). --- @@ -141,7 +141,7 @@ How the input combinations behave: The analysis creates a SQLite database that can be explored using tools like [DB Browser for SQLite](https://sqlitebrowser.org/) or the command line `sqlite3` tool. -**Refer to the [Analyzer documentation](analyzer.md) for the database schema reference and information about extending this command.** +**Refer to the [Analyzer Database Schema](analyzer-schema.md) for the tables and views, and to the [Analyzer documentation](analyzer.md) for information about extending this command.** --- @@ -172,7 +172,7 @@ System.ArgumentException: Invalid object id. This error occurs when SerializedFiles are built without TypeTrees. The command will skip these files and continue. **Solutions:** -- Enable **ForceAlwaysWriteTypeTrees** in your Unity build settings. See [Unity Content Format](../../Documentation/unity-content-format.md) for details. +- Enable **ForceAlwaysWriteTypeTrees** in your Unity build settings. See [Unity Content Format](unity-content-format.md) for details. - If your bundles were built with external TypeTree data (Unity 6.5+), use the `--typetree-data` option to load the TypeTree data file before analysis: ```bash @@ -234,6 +234,6 @@ When `--skip-references` is used, some functionality is lost: When `--skip-crc` is used, the `objects.crc32` column will be 0 for all objects. This means: -* No detection of identical objects by content hash (See [Comparing Builds](../../Documentation/comparing-builds.md)) +* No detection of identical objects by content hash (See [Comparing Builds](comparing-builds.md)) * The `view_potential_duplicates` view relies partially on CRC32 to distinguish true duplicates diff --git a/Documentation/contentlayout-database.md b/Documentation/contentlayout-database.md index 5a8e8ac..b7217c3 100644 --- a/Documentation/contentlayout-database.md +++ b/Documentation/contentlayout-database.md @@ -2,7 +2,7 @@ When the input of the [`analyze`](command-analyze.md) command includes the `ContentLayout.json` of a content directory build, its content is imported into the database tables and views documented on this page. This makes the layout of the build queryable with SQL — which source assets each file was built from, the dependencies between the files, the loadable objects and scenes, and the size of every artifact — and connects that information to the objects and references that `analyze` extracts from the build content itself. -The layout is also what makes reference information correct for a ContentDirectory build: the external reference tables inside the build's files hold symbolic placeholders, and analyze resolves them through the layout's dependency lists. With the layout on the input, references between content files land in the regular `refs` table (and [`find-refs`](command-find-refs.md) works across files); without it they are recorded in [`dangling_refs`](analyzer.md#dangling_refs--dangling_refs_view). +The layout is also what makes reference information correct for a ContentDirectory build: the external reference tables inside the build's files hold symbolic placeholders, and analyze resolves them through the layout's dependency lists. With the layout on the input, references between content files land in the regular `refs` table (and [`find-refs`](command-find-refs.md) works across files); without it they are recorded in [`dangling_refs`](analyzer-schema.md#dangling_refs). For what `ContentLayout.json` contains conceptually, see [ContentLayout.json](contentlayout.md); for the build output format it describes and the reference-resolution mechanics, see [Content Directory Format](contentdirectory-format.md). For the input combinations analyze accepts (including how the layout is matched to the build), see [the `analyze` command](command-analyze.md#contentdirectory-builds). Example queries: [Example queries for ContentDirectory builds](analyze-examples-contentlayout.md). @@ -14,47 +14,118 @@ For what `ContentLayout.json` contains conceptually, see [ContentLayout.json](co * Two adjustments are made so the data is natural to query: * The json's sentinel values are stored as SQL `NULL`: a `SerializedFile` value of `-1` (an object dropped from the build) and the missing `ContentHash` of built-in entries. * The top-level `RootAssets` list is folded into the `is_root_asset` flag of `content_layout_loadable_objects`. -* The `.sql` files in [`Analyzer/Resources`](../Analyzer/Resources) (`ContentLayout*.sql`) are the authoritative reference for the exact columns, in the same way [`ContentLayout.cs`](../UnityDataModels/ContentLayout.cs) is for the json schema. +* This page is the reference for the columns; [`ContentLayout.cs`](../UnityDataModels/ContentLayout.cs) is the reference for the json schema. A produced database also carries short notes on its non-obvious columns inside the `CREATE` statements, so `sqlite3 Analysis.db ".schema content_layout_serialized_files"` describes them without needing this page. ## Tables ### content_layout -One row identifying the imported layout: `name` (the path of the imported file), `version` (the json schema version) and `build_manifest_hash`. +One row identifying the imported layout. + +| Column | Type | Description | +|---|---|---| +| `id` | INTEGER | Always 0 — a single layout per database. Primary key. | +| `name` | TEXT | Path of the imported `ContentLayout.json`. | +| `version` | INTEGER | Schema version of the json file. | +| `build_manifest_hash` | TEXT | The hash used to match the layout to its build. | ### content_layout_serialized_files -One row per serialized file (`.cf` Content File) of the build, keyed by `file_index`. Records the symbolic `cfid` reference string, the `is_builtin` flag, and the `content_hash` that gives the filename (`content_hash || '.cf'`). +One row per serialized file (`.cf` Content File) of the build. -The `serialized_file` column references the core `serialized_files` table, connecting the layout to the analyzed objects (see [below](#analyzing-the-layout-with-or-without-the-build-content) for what the referenced row contains when the build content was not analyzed). +| Column | Type | Description | +|---|---|---| +| `file_index` | INTEGER | The json array index. How every other `content_layout` table names a file. Primary key. | +| `cfid` | TEXT | The symbolic `.cfid` reference string. For built-ins, the built-in path. | +| `is_builtin` | INTEGER | 1 for a built-in entry, which is not a file produced by this build. | +| `content_hash` | TEXT | Gives the filename, `content_hash \|\| '.cf'`. NULL for built-ins. | +| `serialized_file` | INTEGER | [`serialized_files.id`](analyzer-schema.md#serialized_files), connecting the layout to the analyzed objects. NULL for built-ins. See [below](#analyzing-the-layout-with-or-without-the-build-content) for what the referenced row contains when the build content was not analyzed. | ### content_layout_source_assets -The source assets included in each serialized file, one row per (`serialized_file_index`, `asset_path`) pair. The same asset path can appear in more than one file (for example, a single FBX split into multiple output files). +The source assets included in each serialized file. The same asset path can appear in more than one file (for example, a single FBX split into multiple output files), so this is many-to-many. + +| Column | Type | Description | +|---|---|---| +| `serialized_file_index` | INTEGER | References `content_layout_serialized_files.file_index`. | +| `asset_path` | TEXT | Project path of the source asset. | ### content_layout_serialized_file_dependencies -File-to-file dependency edges: the other serialized files that must be loaded before this one (`serialized_file_index` → `dependency_index`). The 1-based `position` column preserves the json array order, which is significant: a PPtr's `m_FileID` inside the file resolves positionally through this list (see [Content Directory Format](contentdirectory-format.md)). +File-to-file dependency edges: the other serialized files that must be loaded before this one. -### content_layout_loadable_dependencies / content_layout_loadable_scene_dependencies +| Column | Type | Description | +|---|---|---| +| `serialized_file_index` | INTEGER | References `content_layout_serialized_files.file_index`. | +| `position` | INTEGER | 1-based position in the json array. Significant: a PPtr's `m_FileID` inside the file resolves positionally through this list (see [Content Directory Format](contentdirectory-format.md)). | +| `dependency_index` | INTEGER | References `content_layout_serialized_files.file_index`. | -The loadables (`object_id_hash`) and scenes (`scene_path`) referenced from each serialized file. +`serialized_file_index` and `position` together are the primary key. ### content_layout_loadable_objects -The objects that can be loaded on demand, keyed by `object_id_hash`. Each row records where the object came from in the source project (`guid`, `asset_path`, `lfid`, `identifier_type`) and where it lives in the built content (`serialized_file_index`, `output_lfid`). `is_root_asset` marks the root assets the build was made from. +The objects that can be loaded on demand, identified independently of the serialized file that contains them. Each row also records where the object came from in the source project. + +| Column | Type | Description | +|---|---|---| +| `object_id_hash` | TEXT | Hash of the GUID, LFID and `identifier_type`. Primary key, and how `content_layout_loadable_dependencies` names a loadable. | +| `guid` | TEXT | AssetDatabase GUID of the source asset. | +| `asset_path` | TEXT | Project path of the source asset. | +| `lfid` | INTEGER | Local file id of the object within the source asset. | +| `identifier_type` | INTEGER | Distinguishes the kinds of identifier a loadable can have. | +| `serialized_file_index` | INTEGER | References `content_layout_serialized_files.file_index`. NULL when the object was dropped from the build (json value `-1`, e.g. server-build shader references). | +| `output_lfid` | INTEGER | Local file id of the object in its output serialized file. | +| `is_root_asset` | INTEGER | 1 for the root assets the build was made from. | ### content_layout_loadable_scenes -The scenes exposed as loadable in the build: `guid`, `path` and the `serialized_file_index` of the file containing the scene. +The scenes exposed as loadable in the build. + +| Column | Type | Description | +|---|---|---| +| `guid` | TEXT | AssetDatabase GUID of the scene. Primary key. | +| `path` | TEXT | Project path of the scene. | +| `serialized_file_index` | INTEGER | References `content_layout_serialized_files.file_index`. NULL when dropped from the build. | + +### content_layout_loadable_dependencies + +The loadable objects each serialized file references. + +| Column | Type | Description | +|---|---|---| +| `serialized_file_index` | INTEGER | References `content_layout_serialized_files.file_index`. | +| `object_id_hash` | TEXT | References `content_layout_loadable_objects.object_id_hash`. | + +### content_layout_loadable_scene_dependencies + +The scenes each serialized file references. + +| Column | Type | Description | +|---|---|---| +| `serialized_file_index` | INTEGER | References `content_layout_serialized_files.file_index`. | +| `scene_path` | TEXT | Matches `content_layout_loadable_scenes.path`. | ### content_layout_binary_artifacts -Every artifact of the build output, keyed by `artifact_index`: the serialized files themselves (`category = 'contentfile'`), the streamed texture/mesh data (`'texture'`, `'mesh'` → `.resS` files), the audio/video data (`'audio'`, `'video'` → `.resource` files) and the manifest (`'manifest'`). This is the standard place to find artifact sizes, including the sizes of the serialized files, which no core table records. +Every artifact of the build output. This is the standard place to find artifact sizes, including the sizes of the serialized files, which no core table records. + +| Column | Type | Description | +|---|---|---| +| `artifact_index` | INTEGER | The json array index. Primary key. | +| `content_hash` | TEXT | The on-disk filename is this plus an extension derived from `category`; `content_layout_binary_artifacts_view` adds it. | +| `category` | TEXT | One of `texture`, `mesh` (both `.resS`), `audio`, `video` (both `.resource`), `contentfile` (`.cf`) or `manifest` (`.json`). | +| `size` | INTEGER | Size of the artifact in bytes. | ### content_layout_artifact_references -Direct references between artifacts (`artifact_index` → `referenced_artifact_index`), e.g. a serialized file referencing its data files. References that go through a loadable are not included, and the graph is never cyclical. References to other serialized files are not recorded here either — those are in `content_layout_serialized_file_dependencies`. +Direct references between artifacts, for example a serialized file referencing its data files. References that go through a loadable are not included, and the graph is never cyclical. References to other serialized files are not recorded here either — those are in `content_layout_serialized_file_dependencies`. + +| Column | Type | Description | +|---|---|---| +| `artifact_index` | INTEGER | References `content_layout_binary_artifacts.artifact_index`. | +| `referenced_artifact_index` | INTEGER | References `content_layout_binary_artifacts.artifact_index`. | + +Both columns together are the primary key. ## Views @@ -76,6 +147,8 @@ SELECT category, COUNT(*) AS count, SUM(size) AS bytes FROM content_layout_binary_artifacts GROUP BY category ORDER BY bytes DESC; ``` +Indexes are created after the tables are populated, so importing a very large layout stays fast. The `content_hash` and `asset_path` indexes carry the views; the rest serve reverse lookups such as "who depends on X" and "which loadables live in file Y". + ## Analyzing the layout with or without the build content A `ContentLayout.json` can be analyzed on its own — useful for running SQL queries against a large layout — or together with the build output it describes. The layout tables themselves are identical in both cases; what differs is what the `serialized_file` link points at: @@ -92,4 +165,4 @@ A `ContentLayout.json` can be analyzed on its own — useful for running SQL que | [Content Directory Format](contentdirectory-format.md) | Content directory builds and inspecting them with UnityDataTool. | | [`analyze` command](command-analyze.md) | The command that imports the layout, and the accepted input combinations. | | [Example queries for ContentDirectory builds](analyze-examples-contentlayout.md) | Example SQL queries against these tables. | -| [Analyzer](analyzer.md) | The core database schema (objects, serialized files, references). | +| [Analyzer Database Schema](analyzer-schema.md) | The core database schema (objects, serialized files, references). | diff --git a/Documentation/unitydatatool.md b/Documentation/unitydatatool.md index e46c6e9..b4f73c0 100644 --- a/Documentation/unitydatatool.md +++ b/Documentation/unitydatatool.md @@ -93,7 +93,8 @@ If you see a warning about `UnityFileSystemApi.dylib` not being verified, go to | Topic | Description | |-------|-------------| -| [Analyzer Database Reference](analyzer.md) | SQLite schema, views, and extending the analyzer | +| [Analyzer Database Schema](analyzer-schema.md) | Reference for the tables and views | +| [Analyzer Library](analyzer.md) | Using and extending the analyzer | | [TextDumper Output Format](textdumper.md) | Understanding dump output | | [ReferenceFinder Details](referencefinder.md) | Reference chain output format | | [Analyze Examples](analyze-examples.md) | Practical database queries | diff --git a/README.md b/README.md index 8d4bcc8..57452fc 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,8 @@ New to Unity's data files or to UnityDataTool? These topics are a good place to | Topic | Description | | --- | --- | | [Command-line tool](./Documentation/unitydatatool.md) | All commands and their options. | -| [Analyzer & database schema](./Documentation/analyzer.md) | The SQLite database that `analyze` produces, including its tables and views. | +| [Analyzer](./Documentation/analyzer.md) | The library behind `analyze`, and where each part of the database is documented. | +| [Database schema](./Documentation/analyzer-schema.md) | Reference for the tables and views that `analyze` produces. | | [Example queries](./Documentation/analyze-examples.md) | Worked examples of querying the analyze database. | | [Using UnityDataTool with an AI agent](./Documentation/agent-guide.md) | The recommended workflow for AI agents (and scripts) analyzing a build. | | [Comparing builds](./Documentation/comparing-builds.md) | Finding what changed between two builds. |