[#120] Move the schema documentation to where it is useful

AGENTS.md

@@ -126,7 +126,7 @@ The feature libraries each build on UnityBinaryFormat and UnityFileSystem. See t
 
 **Handlers**: Type-specific handlers extract specialized properties for Unity object types and populate additional tables. For example Mesh, AnimationClip, Shader, BuildReport, MonoScript.
 
-**Views**: The database schema includes convenient views for seeing the data in useful ways, e.g. `object_view`. See `Documentation/analyzer.md` and `Documentation/addressables-build-reports.md` for schema details.
+**Views**: The database schema includes convenient views for seeing the data in useful ways, e.g. `object_view`. See `Documentation/analyzer-schema.md` for the core schema, and `Documentation/contentlayout-database.md`, `Documentation/buildreport.md` and `Documentation/addressables-build-reports.md` for the parts documented on their own pages.
 
 CLI entry point is `UnityDataTool/Program.cs` using System.CommandLine. Per-command documentation is in `Documentation/`.
 
@@ -135,9 +135,24 @@ CLI entry point is `UnityDataTool/Program.cs` using System.CommandLine. Per-comm
 ### Extending Analyze
 
 * New Unity types can be added by following the same pattern as the existing types, for example MonoScripts.
-* Any database schema change (new or changed tables, views, or columns) must bump `PRAGMA user_version` in `Analyzer/Resources/Init.sql` and extend the version-history comment above it.
+* Any database schema change (new or changed tables, views, or columns) must bump `PRAGMA user_version` in `Analyzer/Resources/Init.sql` and add a row to the version table in `Documentation/analyzer-schema.md`.
 * Analysis of additional file formats could be added, for example AssetBundle manifest files by following the pattern of Addressables build layout files are handled.
 
+#### Commenting the .sql resources
+
+SQLite only keeps the text of the `CREATE` statement itself in `sqlite_master`, so a comment placed
+*above* a statement is discarded and never reaches a produced database. Comments therefore go inside
+the statement:
+
+* One short note on the first line inside the `CREATE TABLE ( ... )` parentheses saying what a row
+  represents, and a trailing `--` note on a column only where the fact is needed to write a correct
+  query and is not guessable from the column name (a foreign key, a sentinel like `''`, an option
+  that leaves the column empty). Views get one purpose line as the first line of the body, after `AS`.
+* Full prose belongs in the documentation, not in the `.sql` file.
+* `CREATE INDEX` has no body, so the only comments left outside a statement are ones that explain the
+  code rather than the schema, such as why the ContentLayout indexes are created after population.
+* Keep `.sql` comments ASCII-only.
+
 ### Other Extensions
 
 The UnityFileSystem API and UnityBinaryFormat parsing can be useful for other analysis.  The "dump", "analyze" and "serialized-file" commands can be considered reference examples of how to use those lower level tools.

Analyzer/Properties/Resources.resx

@@ -131,10 +131,10 @@
     <value>..\Resources\BuildReport.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8</value>
   </data>
   <data name="Finalize" type="System.Resources.ResXFileRef, System.Windows.Forms">
-    <value>..\Resources\Finalize.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;Windows-1252</value>
+    <value>..\Resources\Finalize.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8</value>
   </data>
   <data name="Init" type="System.Resources.ResXFileRef, System.Windows.Forms">
-    <value>..\Resources\Init.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;Windows-1252</value>
+    <value>..\Resources\Init.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8</value>
   </data>
   <data name="Mesh" type="System.Resources.ResXFileRef, System.Windows.Forms">
     <value>..\Resources\Mesh.sql;System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089;utf-8</value>

Analyzer/Resources/AssetBundle.sql

@@ -1,39 +1,28 @@
--- tables related to the AssetBundle and PreloadData objects
-
--- Do not confuse the AssetBundle Unity object (the source of much of this data)
--- with the archives table, which is general to any Unity Archive.
-
--- The "assets" that an AssetBundle explicitly exposes: each m_Container entry of the AssetBundle
--- object names an object (the addressable/asset name -> object it maps to). Populated only from
--- the AssetBundle object, so this table is empty for Player and ContentDirectory builds.
--- For scene bundles the entry names the scene and points at the synthetic Scene object (see
--- AssetBundleHandler / SerializedFileSQLiteWriter).
 CREATE TABLE IF NOT EXISTS assetbundle_assets(
-    object INTEGER,
-    name TEXT
+    -- The assets an AssetBundle explicitly exposes: one row per m_Container entry of the
+    -- AssetBundle object. Empty for Player and ContentDirectory builds, which have no such object.
+    object INTEGER,     -- objects.id; for a scene bundle this is the synthetic Scene object
+    name TEXT           -- the container path the asset is addressed by
 );
 
--- object depends on dependency. This table has three sources, only the first of which is truly
--- AssetBundle-specific:
---   * AssetBundleHandler: an asset's slice of the AssetBundle object's m_PreloadTable.
---   * SerializedFileSQLiteWriter: a scene object -> each object in the scene's SerializedFiles.
---   * PreloadDataHandler: the PreloadData object's m_Assets. PreloadData is a *separate* Unity
---     object (not part of the AssetBundle object) and also exists in Player builds (one per scene
---     in its sharedAssetsN.assets, plus one in globalgamemanagers.assets), so this table is NOT
---     empty there. Player builds have no scene object, so those rows hang off the PreloadData
---     object itself; scene bundles hang them off the synthetic Scene object.
 CREATE TABLE IF NOT EXISTS preload_dependencies(
-    object INTEGER,
-    dependency INTEGER
+    -- Objects that Unity preloads alongside another object. Populated for AssetBundle and Player
+    -- builds, but not ContentDirectory builds. See Documentation/analyzer-schema.md.
+    object INTEGER,     -- objects.id: an AssetBundle asset, a synthetic Scene, or a PreloadData object
+    dependency INTEGER  -- objects.id, or dangling_refs.id when the target was not analyzed
 );
 
 CREATE VIEW IF NOT EXISTS assetbundle_asset_view AS
+-- AssetBundle assets with their object columns resolved. Inner join, so an asset whose object was
+-- not analyzed is omitted here but still present in assetbundle_assets.
 SELECT
     a.name AS asset_name,
     o.*
 FROM assetbundle_assets a INNER JOIN object_view o ON o.id = a.object;
 
 CREATE VIEW IF NOT EXISTS preload_dependencies_view AS
+-- Preload dependencies of AssetBundle assets and scenes, with both sides resolved. Narrower than
+-- the table: Player-build rows and dangling dependencies drop out of the inner joins.
 SELECT a.id, a.asset_name, a.archive, a.type, od.id dep_id, od.archive dep_archive, od.name dep_name, od.type dep_type
 FROM assetbundle_asset_view a
 INNER JOIN preload_dependencies d ON a.id = d.object

Analyzer/Resources/ContentLayout.sql

@@ -1,8 +1,7 @@
--- Identity of the imported ContentLayout.json (see Documentation/contentlayout.md). The
--- content_layout* tables are only created when a ContentLayout.json is part of the analyzed
--- input. A single layout per database is supported.
 CREATE TABLE IF NOT EXISTS content_layout
 (
+    -- Identity of the imported ContentLayout.json. The content_layout* tables exist only when a
+    -- layout was part of the analyzed input. See Documentation/contentlayout-database.md.
     id INTEGER,                 -- always 0 (single layout per database)
     name TEXT,                  -- path of the imported ContentLayout.json
     version INTEGER,            -- schema version of the json file

Analyzer/Resources/ContentLayoutArtifactReferences.sql

@@ -1,9 +1,8 @@
--- Direct references between binary artifacts (ArtifactReferences in the json), e.g. a
--- serialized file referencing its .resS/.resource data files. References that go through a
--- loadable are not included, and the graph is never cyclical. References to other serialized
--- files are not recorded here either; those are in content_layout_serialized_file_dependencies.
 CREATE TABLE IF NOT EXISTS content_layout_artifact_references
 (
+    -- Direct references between binary artifacts, e.g. a content file to its .resS/.resource data
+    -- files. Never cyclical. Content-file-to-content-file edges live in
+    -- content_layout_serialized_file_dependencies instead.
     artifact_index INTEGER,           -- references content_layout_binary_artifacts.artifact_index
     referenced_artifact_index INTEGER,
     PRIMARY KEY (artifact_index, referenced_artifact_index)

Analyzer/Resources/ContentLayoutBinaryArtifacts.sql

@@ -1,11 +1,9 @@
--- The artifacts that make up the build output (BinaryArtifacts in the json): the serialized
--- files themselves plus the data files they use (.resS, .resource) and the manifest. This is the
--- standard place to find artifact sizes. When stored as a file the filename is the content hash
--- plus an extension derived from the category (content_layout_binary_artifacts_view adds it).
 CREATE TABLE IF NOT EXISTS content_layout_binary_artifacts
 (
+    -- Every artifact making up the build output: the content files plus the data files they use
+    -- (.resS, .resource) and the manifest. The standard place to find artifact sizes.
     artifact_index INTEGER,
-    content_hash TEXT,
+    content_hash TEXT,          -- the on-disk filename is content_hash plus an extension from category
     category TEXT,              -- 'texture' | 'mesh' | 'audio' | 'video' | 'contentfile' | 'manifest'
     size INTEGER,
     PRIMARY KEY (artifact_index)

Analyzer/Resources/ContentLayoutIndexes.sql

@@ -1,6 +1,4 @@
--- Created after the content_layout tables are populated, so inserts stay fast for very large
--- layouts. The content_hash and asset_path indexes carry the views; the rest serve reverse
--- lookups ("who depends on X", "which loadables live in file Y").
+-- Created after the content_layout tables are populated so inserts stay fast for very large layouts.
 CREATE INDEX content_layout_binary_artifacts_content_hash ON content_layout_binary_artifacts(content_hash);
 CREATE INDEX content_layout_source_assets_asset_path ON content_layout_source_assets(asset_path);
 CREATE INDEX content_layout_source_assets_file ON content_layout_source_assets(serialized_file_index);

Analyzer/Resources/ContentLayoutLoadableDependencies.sql

@@ -1,6 +1,6 @@
--- Loadable objects referenced from each serialized file (LoadableDependencies in the json).
 CREATE TABLE IF NOT EXISTS content_layout_loadable_dependencies
 (
+    -- The loadable objects each content file references.
     serialized_file_index INTEGER,   -- references content_layout_serialized_files.file_index
     object_id_hash TEXT              -- references content_layout_loadable_objects.object_id_hash
 );

Analyzer/Resources/ContentLayoutLoadableObjects.sql

@@ -1,17 +1,14 @@
--- The objects that can be loaded on demand (LoadableObjectIds in the json), identified
--- independently of the serialized file that contains them. Also records where each one came from
--- in the source project. The json's top-level RootAssets list is folded into the is_root_asset
--- flag. serialized_file_index is NULL when the object was dropped from the build (json value -1,
--- e.g. server build shader references).
 CREATE TABLE IF NOT EXISTS content_layout_loadable_objects
 (
+    -- The objects that can be loaded on demand, identified independently of the content file that
+    -- holds them, plus where each came from in the source project.
     object_id_hash TEXT,             -- hash of GUID, LFID and identifier_type
     guid TEXT,                       -- AssetDatabase GUID of the source asset
     asset_path TEXT,
     lfid INTEGER,                    -- local file id of the object in the source asset
     identifier_type INTEGER,
-    serialized_file_index INTEGER,   -- references content_layout_serialized_files.file_index, or NULL
-    output_lfid INTEGER,             -- local file id of the object in its output serialized file
+    serialized_file_index INTEGER,   -- content_layout_serialized_files.file_index; NULL if dropped from the build
+    output_lfid INTEGER,             -- local file id of the object in its output content file
     is_root_asset INTEGER,
     PRIMARY KEY (object_id_hash)
 );

Analyzer/Resources/ContentLayoutLoadableSceneDependencies.sql

@@ -1,6 +1,6 @@
--- Scenes referenced from each serialized file (LoadableSceneDependencies in the json).
 CREATE TABLE IF NOT EXISTS content_layout_loadable_scene_dependencies
 (
+    -- The scenes each content file references.
     serialized_file_index INTEGER,   -- references content_layout_serialized_files.file_index
     scene_path TEXT                  -- matches content_layout_loadable_scenes.path
 );

Analyzer/Resources/ContentLayoutLoadableScenes.sql

@@ -1,8 +1,8 @@
--- The scenes exposed as loadable in the build (LoadableSceneIds in the json).
 CREATE TABLE IF NOT EXISTS content_layout_loadable_scenes
 (
+    -- The scenes exposed as loadable in the build.
     guid TEXT,
     path TEXT,
-    serialized_file_index INTEGER,   -- references content_layout_serialized_files.file_index, or NULL
+    serialized_file_index INTEGER,   -- content_layout_serialized_files.file_index; NULL if dropped from the build
     PRIMARY KEY (guid)
 );

Analyzer/Resources/ContentLayoutSerializedFileDependencies.sql

@@ -1,11 +1,8 @@
--- File-to-file dependency edges (SerializedFileDependencies in the json): the other serialized
--- files that must be loaded before this one. position preserves the array order, which is
--- significant: a PPtr's m_FileID inside the file resolves positionally through this list (see
--- Documentation/contentdirectory-format.md).
 CREATE TABLE IF NOT EXISTS content_layout_serialized_file_dependencies
 (
+    -- File-to-file dependency edges: the other content files that must be loaded before this one.
     serialized_file_index INTEGER,   -- references content_layout_serialized_files.file_index
-    position INTEGER,                -- 1-based, matching the external reference table / m_FileID index
+    position INTEGER,                -- 1-based; a PPtr's m_FileID resolves positionally through this list
     dependency_index INTEGER,        -- references content_layout_serialized_files.file_index
     PRIMARY KEY (serialized_file_index, position)
 );

Analyzer/Resources/ContentLayoutSerializedFiles.sql

@@ -1,15 +1,11 @@
--- One row per entry in the layout's SerializedFiles array: the serialized files (.cf Content
--- Files) that the ContentDirectory build produced. file_index is the array index from the json
--- and is how the other content_layout tables reference a file.
--- serialized_file links to the core serialized_files table so layout data joins directly with
--- analyzed objects and references; it is NULL when the analyzed input did not include the build
--- content (e.g. a layout-only analyze) or for built-in entries.
 CREATE TABLE IF NOT EXISTS content_layout_serialized_files
 (
-    file_index INTEGER,
+    -- The content files (.cf) the ContentDirectory build produced, one row per entry in the
+    -- layout's SerializedFiles array.
+    file_index INTEGER,         -- the json array index; how the other content_layout tables name a file
     cfid TEXT,                  -- symbolic .cfid reference string (for built-ins: the built-in path)
     is_builtin INTEGER,
     content_hash TEXT,          -- NULL for built-ins; the filename is content_hash || '.cf'
-    serialized_file INTEGER,    -- references serialized_files.id, or NULL
+    serialized_file INTEGER,    -- serialized_files.id; NULL for built-ins and for a layout-only analyze
     PRIMARY KEY (file_index)
 );

Analyzer/Resources/ContentLayoutSourceAssets.sql

@@ -1,7 +1,7 @@
--- The source assets included in each serialized file (SourceAssets in the json). The same asset
--- path can appear in more than one file (e.g. an FBX split into multiple output files).
 CREATE TABLE IF NOT EXISTS content_layout_source_assets
 (
+    -- The source assets built into each content file. The same asset path can appear in more than
+    -- one file, so this is many-to-many.
     serialized_file_index INTEGER,   -- references content_layout_serialized_files.file_index
     asset_path TEXT
 );

Analyzer/Resources/ContentLayoutViews.sql

@@ -1,10 +1,6 @@
--- Convenience views over the content_layout tables. Created together with the tables (only when
--- a ContentLayout.json is imported). See Documentation/contentlayout-database.md for the schema
--- reference.
-
--- One row per layout serialized file with the derived filename, artifact size, and core-table link.
--- Built-in entries have no file on disk, so their path (cfid) is shown as the filename.
 CREATE VIEW IF NOT EXISTS content_layout_serialized_files_view AS
+-- One row per layout content file with its derived filename, artifact size and core-table link.
+-- Built-in entries have no file on disk, so their path (cfid) is shown as the filename.
 SELECT f.file_index, f.cfid, f.is_builtin,
        CASE WHEN f.is_builtin = 1 THEN f.cfid ELSE f.content_hash || '.cf' END AS filename,
        ba.size,
@@ -14,24 +10,22 @@ FROM content_layout_serialized_files f
 LEFT JOIN content_layout_binary_artifacts ba ON ba.content_hash = f.content_hash AND ba.category = 'contentfile'
 LEFT JOIN serialized_files sf ON sf.id = f.serialized_file;
 
--- Source asset -> the file(s) it was built into. Built-in entries have no source assets, so no
--- filename fallback is needed here.
 CREATE VIEW IF NOT EXISTS content_layout_source_assets_view AS
+-- Source asset to the content file(s) it was built into.
 SELECT s.asset_path, f.file_index, f.content_hash || '.cf' AS filename, f.serialized_file
 FROM content_layout_source_assets s
 INNER JOIN content_layout_serialized_files f ON f.file_index = s.serialized_file_index;
 
--- File-to-file dependency edges with names resolved on both sides (via the files view, so
--- dependencies on built-in entries show their path instead of a NULL filename).
 CREATE VIEW IF NOT EXISTS content_layout_serialized_file_dependencies_view AS
+-- File-to-file dependency edges with filenames resolved on both sides.
 SELECT d.serialized_file_index, src.filename, d.position,
        d.dependency_index, dep.filename AS dependency_filename, dep.cfid AS dependency_cfid
 FROM content_layout_serialized_file_dependencies d
 INNER JOIN content_layout_serialized_files_view src ON src.file_index = d.serialized_file_index
 INNER JOIN content_layout_serialized_files_view dep ON dep.file_index = d.dependency_index;
 
--- Loadables resolved to their analyzed object (object columns are NULL in a layout-only database).
 CREATE VIEW IF NOT EXISTS content_layout_loadable_objects_view AS
+-- Loadables resolved to their analyzed object. The object columns are NULL in a layout-only database.
 SELECT l.object_id_hash, l.guid, l.asset_path, l.lfid, l.is_root_asset,
        f.content_hash || '.cf' AS filename,
        o.id AS object, t.name AS type, o.name, o.size
@@ -40,8 +34,8 @@ LEFT JOIN content_layout_serialized_files f ON f.file_index = l.serialized_file_
 LEFT JOIN objects o ON o.serialized_file = f.serialized_file AND o.object_id = l.output_lfid
 LEFT JOIN types t ON t.id = o.type;
 
--- Artifacts with their derived on-disk filename (extension based on the category).
 CREATE VIEW IF NOT EXISTS content_layout_binary_artifacts_view AS
+-- Artifacts with their on-disk filename derived from the content hash and category.
 SELECT artifact_index, content_hash, category, size,
        content_hash ||
        CASE category
@@ -52,8 +46,8 @@ SELECT artifact_index, content_hash, category, size,
        END AS filename
 FROM content_layout_binary_artifacts;
 
--- The data files (.resS/.resource) each serialized file uses, derived from the artifact graph.
 CREATE VIEW IF NOT EXISTS content_layout_resource_files_view AS
+-- The data files (.resS/.resource) each content file uses, derived from the artifact graph.
 SELECT f.file_index, f.content_hash || '.cf' AS filename,
        ra.category, rav.filename AS data_filename, ra.size
 FROM content_layout_serialized_files f