Skip to content

[#120] Move the schema documentation to where it is useful - #136

Merged
SkowronskiAndrew merged 1 commit into
mainfrom
issue120-sql-comments
Sep 15, 2026
Merged

SkowronskiAndrew merged 1 commit into
mainfrom
issue120-sql-comments

Conversation

@SkowronskiAndrew

Copy link
Copy Markdown
Collaborator

Summary

Fixes #120.

SQLite only keeps the text of the CREATE statement itself in sqlite_master, so a comment placed
above a statement is discarded. All 58 comment lines documenting the core schema in Init.sql
were above their statements, so none of them reached a produced database: .schema objects showed
bare column names, and anyone working from a .db artifact never saw the documentation.

This takes both halves of the fix offered in the issue. Concise key facts move inside the CREATE
statements, where SQLite preserves them verbatim, so a produced database is self-describing. The
detailed prose moves into the documentation, which has shipped in the release zip since #118, so it
travels with the binary. Every comment outside a CREATE is gone, so the old practice stops rather
than being repeated by the next table.

Only facts that are needed to write a correct query and are not guessable from the column name went
into the schema, for example that serialized_files.archive and objects.game_object hold ''
rather than NULL, that refs.referenced_object can be a dangling_refs.id, and that --skip-crc
sets every crc32 to 0. .schema serialized_files now reads:

CREATE TABLE serialized_files
(
    -- One row per SerializedFile analyzed.
    id INTEGER,
    archive INTEGER,    -- archives.id, or '' when the file is not inside an archive
    name TEXT,          -- on-disk name, usually hash-based: CAB-<hash>, BuildPlayer-<scene>, levelN
    PRIMARY KEY (id)
);

No table, view or column changed — verified mechanically that all 15 changed .sql files are
byte-identical to their previous versions once comments are stripped. PRAGMA user_version therefore
stays at 7; bumping it would falsely signal an incompatibility to find-refs.

Changes

SQL resources

  • Init.sql: all 58 comments outside CREATE statements removed. Table-level notes and per-column
    notes added inside the parentheses; each of the 7 views gained a one-line purpose comment as the
    first line of its body (5 of them had no comment at all before). The INSERT INTO types comment
    could not survive on its own, so the -1 Scene sentinel is documented on types.id instead.
  • AssetBundle.sql had the identical bug — 19 lines of good comments, all discarded. Same treatment.
  • The 13 ContentLayout*.sql files were half converted: good in-paren column notes plus a prose
    block above each statement that was thrown away. The prose blocks are gone and anything
    query-critical that lived only there (e.g. why serialized_file is NULL for a layout-only
    analyze) was promoted into the parentheses.
  • ContentLayoutViews.sql and ContentSummary.sql: per-view comments moved inside the view bodies.
  • ContentLayoutIndexes.sql keeps one comment outside the statements, trimmed to the "why late"
    rationale. CREATE INDEX has no body to hold a note, and that comment explains the code rather
    than the schema.
  • AGENTS.md records the convention so new .sql files follow it.

Documentation

  • New Documentation/analyzer-schema.md: the schema reference, in one uniform format
    (| Column | Type | Description | per entity, plus an at-a-glance index). Replaces the four
    different styles previously in use across the schema pages.
  • Documentation/analyzer.md was 81% schema reference; it is now a 76-line overview (intro, a map of
    where each part of the schema is documented, and the Advanced/extending sections).
  • Documentation/contentlayout-database.md reformatted to the same markdown-table shape, absorbing
    the prose evicted from the ContentLayout*.sql files.
  • serialized_files, types, property_names and property_types gained documentation they never
    had — the SerializedFile naming rules per build pipeline (CAB-, BuildPlayer-, levelN) existed
    only in a discarded Init.sql comment.
  • The schema version history moved out of the Init.sql comment into a markdown table on the new
    page; AGENTS.md points the bump rule at it.
  • agent-guide.md told agents that "the produced database's schema shows bare table definitions",
    which this change makes false. Rewritten.
  • Referrers retargeted so links promising a schema reference land on the new page.

Incidental fixes

  • Analyzer/Properties/Resources.resx: Init.sql and Finalize.sql were declared Windows-1252
    while the other 49 .sql entries are utf-8. Both date from the initial commit and there is no
    purpose to it; all .sql files are pure ASCII, which decodes identically under either encoding, so
    the switch to utf-8 is a no-op on current content and removes a trap for future comment edits.
  • The /* TABLE DEFINITION: */ blocks in the nine Commands/SerializedFile/*.cs classes hand-copied
    the DDL and had to be kept in sync manually. They are replaced with a one-line pointer to the
    owning .sql file. The drift was already real: AddPreloadDependency declared
    PRIMARY KEY (object, dependency), which the actual DDL has never had.
  • Nine pre-existing broken documentation links fixed, found by a link check written for this PR.
    Notably analyzer.md described an IWriter interface that no longer exists in the codebase.

Testing

  • dotnet build -c Release — clean, 0 warnings.
  • dotnet test -c Release — full suite green: 787 passed, 0 failed (Analyzer.Tests 62,
    UnityFileSystem.Tests 435, UnityDataTool.Tests 290). Any .sql syntax error fails essentially
    every analyze test at Begin(), so this is the real regression net.
  • Manual verification on a real database: ran analyze against TestCommon/Data/AssetBundles and
    against the ContentDirectory build in TestCommon/Data/LeadingEdgeBuilds, then confirmed with
    sqlite3 ... ".schema <table>" that the notes are present and readable for core tables,
    AssetBundle tables, ContentLayout tables and views; that queries against object_view and
    view_breakdown_by_type still work; and that PRAGMA user_version still reports 7. Re-ran after
    the ResX encoding change to confirm the resource still loads.
  • Verified all 15 changed .sql files are byte-identical ignoring comments, so the schema itself is
    untouched.
  • Verified every internal documentation link and anchor across all 26 markdown files resolves
    (0 problems), since there is no CI link checker.
  • Verified all .sql files are pure ASCII and that the only remaining comment outside a CREATE
    statement is the intended ContentLayoutIndexes.sql one.

🤖 Generated with Claude Code

SQLite only keeps the text of the CREATE statement itself in sqlite_master, so
the 58 comment lines documenting the schema in Init.sql reached nobody: .schema
on a produced database showed bare column names.

Comments now go inside the CREATE statements, where SQLite preserves them, so a
produced .db is self-describing. Only facts needed to write a correct query and
not guessable from the column name are kept there; the prose moved into the
documentation, which ships in the release zip since #118.

The same fix applies to AssetBundle.sql, the ContentLayout*.sql files (which
already had good in-paren column notes plus a discarded prose block above each
statement) and ContentSummary.sql. AGENTS.md records the convention.

Documentation/analyzer.md was 81% schema reference, so the schema moved to a new
strictly formatted Documentation/analyzer-schema.md and analyzer.md became the
overview. contentlayout-database.md gained the same markdown-table format. The
schema version history moved out of the Init.sql comment into a table on the new
page, and serialized_files, types, property_names and property_types gained
documentation they never had.

No table, view or column changed, so PRAGMA user_version stays at 7.

Also fixed while here: the ResX file-reference encoding for Init.sql and
Finalize.sql (Windows-1252 since the initial commit, every other .sql is utf-8),
the duplicated DDL in the Commands/SerializedFile classes (AddPreloadDependency
declared a primary key the real DDL never had), and nine pre-existing broken
documentation links.
@SkowronskiAndrew
SkowronskiAndrew merged commit d026163 into main Sep 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Analyzer: Init.sql schema comments never reach the produced database

1 participant