ADFA-5153: Migration script for the shared Brotli dictionary

.gitignore

@@ -198,6 +198,10 @@ TEST_*.md
 assets-*.zip
 dynamic_libs/*.aar.br
 
+# Python bytecode from scripts/
+__pycache__/
+*.pyc
+
 # Per-project cache the IDE writes (models, sync metadata, locks). The test project's copy was
 # tracked and every test run rewrote it with the local machine's absolute paths, so it arrived in
 # unrelated commits -- a 12 MB binary among them (ADFA-5264).

build.gradle.kts

@@ -335,7 +335,18 @@ spotless {
 			".githooks/**/*",
 			"scripts/**/*",
 		)
-		targetExclude("scripts/debug-keystore/adfa-keystore.jks")
+		targetExclude(
+			"scripts/debug-keystore/adfa-keystore.jks",
+			// leadingSpacesToTabs() would reindent Python, which PEP 8 indents with spaces --
+			// as all but one .py here already is (scripts/r8-plugin-impact/analyze-plugin-impact.py
+			// uses tabs). Only the ratchet has been hiding the mismatch: an edit to a
+			// space-indented one would silently convert the whole file.
+			"**/*.py",
+			// Python bytecode: binary, generated, and Spotless fails the whole task (and so the
+			// pre-push hook) on one stray file rather than skipping it.
+			"**/__pycache__/**",
+			"**/*.pyc",
+		)
 	}
 }
 

docs/documentation-database.md

@@ -34,8 +34,9 @@ CREATE TABLE Content (
 One row per file the web server can serve (HTML, CSS, JS, image, video, PDF, ...) — 30,000+ rows. Key points:
 
 - **`path`** is the lookup key (indexed via the `UNIQUE` constraint) and is what `WebServer` matches the HTTP request path against. Paths carry a short source prefix to avoid collisions between doc sets, e.g. `k/index.html` (Kotlin) vs `j/index.html` (Java).
-- **`content`** is compressed — Brotli for text-like formats, format-specific compression otherwise (images/video/fonts). `ContentTypes.compression` says which. Every migrated `Content` row with `ContentTypes.compression = 'brotli'` is Brotli-compressed against the single shared dictionary in `CompressionDictionary` (see below), converted in one pass by ADFA-5153 — but plugin-contributed Tier 3 rows (`PluginDocumentationManager`/`BrotliCompressor`, see below) are plain, dictionary-free Brotli, and there is no per-row flag distinguishing the two, because a dictionary-compressed stream and a plain one are not distinguishable at decode time by inspection. They *are* distinguishable by attempting the decode: attaching the *wrong* dictionary decodes without error to different bytes than were compressed (its backward distances resolve into real, just incorrect, bytes) — but attaching *no* dictionary to a stream that needs one reliably throws (`IOException`, "corrupted input"), since distances into the dictionary region are then out of bounds for any spec-compliant decoder. `DocumentationContentSource` (which both Tier 3 transports read through) relies on exactly this: its decode tries the dictionary first and falls back to a plain decode on `IOException`, which correctly handles both dictionary-compressed and plain rows — but never rely on decode success/failure to detect a *wrong* dictionary, since that case is silent. Content over 1 MB is split across multiple rows: the first row's path is the base path, continuation rows are `path-1`, `path-2`, ... (`languageId = 1`), reassembled by `DocumentationContentSource` before returning.
+- **`content`** is compressed — Brotli for text-like formats, format-specific compression otherwise (images/video/fonts). `ContentTypes.compression` says which. Every migrated `Content` row with `ContentTypes.compression = 'brotli'` is Brotli-compressed against the single shared dictionary in `CompressionDictionary` (see below), converted in one pass by ADFA-5153 — but plugin-contributed Tier 3 rows (`PluginDocumentationManager`/`BrotliCompressor`, see below) are plain, dictionary-free Brotli, and there is no per-row flag distinguishing the two, because a dictionary-compressed stream and a plain one are not distinguishable at decode time by inspection. They *are* distinguishable by attempting the decode: attaching the *wrong* dictionary decodes without error to different bytes than were compressed (its backward distances resolve into real, just incorrect, bytes) — but attaching *no* dictionary to a stream that needs one reliably throws (`IOException`, "corrupted input"), since distances into the dictionary region are then out of bounds for any spec-compliant decoder. `DocumentationContentSource` (which both Tier 3 transports read through) relies on exactly this: its decode tries the dictionary first and falls back to a plain decode on `IOException`. That handles the common case, **but the fallback is not a guarantee in either direction**: attaching a dictionary to a stream that never used one can also decode *without error, to different bytes*, in which case nothing throws and the fallback never fires. Measured with brotli 1.2.0: a 92-byte stored stream of repetitive HTML decodes plainly to its real 49,200 bytes and, with a dictionary attached, to 49,200 bytes of garbage with exit 0. Short stored streams are where this lives. So never rely on decode success/failure to detect a wrong dictionary *or* a dictionary/plain mismatch — both are silent — and treat a half-migrated database as able to serve wrong content, not merely as incomplete. Content over 1 MB is split across multiple rows: the first row's path is the base path, continuation rows are `path-1`, `path-2`, ... (`languageId = 1`), reassembled by `DocumentationContentSource` before returning.
 - **`templateId`**: `0` (or unset) means `content` is legacy HTML with presentation baked in (the pre-CMS Release 0/1 format). A positive value means `content` is JSON *facts only*, rendered through the matching row in `Templates` (a Pebble template) — the ongoing move to a proper CMS that de-duplicates presentation across near-identical pages (e.g. `sin`/`cos` docs).
+- Two data defects live in the shipped rows rather than in the schema, and `scripts/docdb/migrate_content_to_dictionary_brotli.py` repairs both before it recompresses anything. **Chunk numbering:** 14 of the 19 chunked items number their continuations from `-2`, not the `-1` the reassembly loop starts at (ADFA-5171), so those items serve as their first 1 MiB and nothing more; the script's `renumber` phase shifts them down. **Mislabelled types:** 74 rows holding GIF/PNG/JPEG/QuickTime payloads are typed `text/plain` (ADFA-5221), so they are Brotli-compressed for no gain and served as `Content-Type: text/plain`; the `retype` phase stores their plaintext and points them at the type their magic bytes prove they are, which -- since those types carry `compression = 'none'` -- also drops them out of the dictionary pass. Both defects originate in `docdb-studio`'s import path, so a freshly exported database will carry them again until fixed there.
 - The `UNIQUE(path)` constraint rejects any duplicate `path`, regardless of `languageID` — a second language for an existing path isn't supported yet (only `EN-us` currently exists). Getting there needs an upstream schema change to composite uniqueness on `(path, languageID)` (see *Known rough edges* below).
 
 Dimensions: `Languages(id, value)` (4-letter codes, e.g. `EN-us`); `ContentTypes(id, value, compression)` (MIME type + compression scheme, ~30 rows).
@@ -63,7 +64,7 @@ CREATE TABLE Tooltips (
 ### Supporting tables
 
 - **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. **Exactly one row**: the version the file *is*, which `populate_db.py` replaces rather than appends. Nothing in the DDL enforces that, so both readers (`DatabaseVersionResolver.resolveMajorVersion` here, `database_major_version` in docdb-studio) take the newest row rather than the one with the highest `major` (the app orders by `changeTime DESC, rowid DESC`, so the greatest `changeTime` wins and `rowid` only breaks ties) — a rebuild from an older content set is a downgrade and has to read as one, which `MAX(major)` would get wrong. The app's reader additionally logs a warning when it finds more than one row; docdb-studio's does not. `populate_db.py` collapses a file that somehow accumulated several back to one, and `resolveMajorVersion` returns null for a database predating the table. `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `DocumentationContentSource` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every row would then pay a failed dictionary decode before its plain one, on every request), and a migrated database that lost the table fails loudly instead of quietly.
-- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `DocumentationContentSource` loads it lazily -- not merely from opening or swapping databases, but on the first content fetch that needs it after the active database changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below).
+- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows, after repairing the two data defects noted above) (never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `DocumentationContentSource` loads it lazily -- not merely from opening or swapping databases, but on the first content fetch that needs it after the active database changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below).
 - **`Templates(id, name, content)`** — Pebble template source, keyed by id (and by `name` for well-known templates like `bookshelf`). Referenced by `Content.templateId`.
 - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`).
 - **`LastChange(documentationSet, changeTime, who)`** — audit trail for edits made through `docdb-studio`; not shown to end users. `DatabaseVersionResolver` reads the `documentationSet = 'wholedb'` row to report the DB's build/edit stamp in debug logging, falling back to the most recent row of any set if `'wholedb'` is missing.