From e0299279a344cf5e01fff5fa5c2616a60aea1d5e Mon Sep 17 00:00:00 2001 From: David Schachter Date: Thu, 20 Aug 2026 16:25:05 -0700 Subject: [PATCH 01/16] ADFA-5153: Add a parallel dictionary-recompression script for documentation.db Recompresses every brotli Content row against the dictionary already in the database's CompressionDictionary table. Written for the 20-Aug database, which has the dictionary but plain-Brotli rows, so nothing benefits from it yet. Two things about the data decided the design, both checked rather than assumed: Content over 1 MiB is not stored as independently compressed pieces. The rows are raw 1 MiB slices of a single Brotli stream -- a slice alone does not decode -- so the unit of work is a base row plus its continuations, concatenated, decoded, recompressed and re-split. A naive per-row migration would have destroyed all three such items, silently, since each slice still looks like a blob. And those continuation rows are numbered from -2 while WebServer's reassembly loop starts at -1 (ADFA-5170), so they already serve truncated. The script preserves whatever numbering it finds, keeping the migration behaviour-neutral; --renumber-continuations rewrites from -1 instead, which makes them reachable again, as an opt-in rather than a side effect. Classification tries the plain decode first, deliberately: attaching no dictionary to a stream that needs one reliably fails, so a successful plain decode proves a row is unmigrated. The reverse is not safe -- a dictionary attached to a stream that never used one can decode to different bytes without erroring. A row that decodes identically both ways is left alone; those are tiny already-compressed payloads the compressor found nothing to reference for. Every item is verified before it is written: the recompressed bytes must decode back to exactly the original plaintext, or the item is reported as an error and left as it was. Measured on a copy of the 20-Aug database, 20 workers: 29,751 items, no errors, 129.0 MiB of stored content down to 85.7 MiB (33.6%), 3.3 minutes against about 73 single-threaded. The file itself goes 313.8 MB to 267.7 MB after VACUUM, and integrity_check passes. Verified independently of the script's own accounting: 303 sampled items, including all three chunked ones, decode with the dictionary to content byte-identical to what the source decodes plainly. Re-running is cheap (0.1 min) and converges -- pass two rewrote one row 11 bytes smaller, passes three and four changed nothing. --- docs/documentation-database.md | 2 +- .../migrate_content_to_dictionary_brotli.py | 387 ++++++++++++++++++ 2 files changed, 388 insertions(+), 1 deletion(-) create mode 100755 scripts/docdb/migrate_content_to_dictionary_brotli.py diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 3055c955f0..355861fdd2 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -63,7 +63,7 @@ CREATE TABLE Tooltips ( ### Supporting tables - **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. Append-only: each change is another `INSERT`, so the **row inserted last** is the current version, not the highest one ever recorded — a rebuild from an older content set is a downgrade and has to read as one (`DatabaseVersionResolver.resolveMajorVersion`, which returns null for a database predating the table). `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `WebServer` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every row would then pay a failed dictionary decode before its plain one, on every request), and a migrated database that lost the table fails loudly instead of quietly. -- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). +- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows)(never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). - **`Templates(id, name, content)`** — Pebble template source, keyed by id (and by `name` for well-known templates like `bookshelf`). Referenced by `Content.templateId`. - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`). - **`LastChange(documentationSet, changeTime, who)`** — audit trail for edits made through `docdb-studio`; not shown to end users. `DatabaseVersionResolver` reads the `documentationSet = 'wholedb'` row to report the DB's build/edit stamp in debug logging, falling back to the most recent row of any set if `'wholedb'` is missing. diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py new file mode 100755 index 0000000000..af0f6dc885 --- /dev/null +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -0,0 +1,387 @@ +#!/usr/bin/env python3 +"""Recompress documentation.db's Brotli Content rows against the shared dictionary. + +Reads the dictionary from the database's own CompressionDictionary table (id = 1) +and rewrites every `ContentTypes.compression = 'brotli'` row so it is compressed +against that dictionary instead of plainly. WebServer tries a dictionary-attached +decode first and falls back to a plain one, so a half-migrated database still +serves -- which is what makes running this incrementally safe. + +Two properties of the data shape this script, both verified against the 20-Aug +database rather than assumed: + + * Content over 1 MiB is *not* stored as independently compressed pieces. The + rows are raw 1 MiB slices of one Brotli stream: `path`, then `path-N` + continuations. A slice on its own does not decode. So the unit of work here + is a logical item -- a base row plus its continuations -- concatenated, + decoded, recompressed, and re-split. Migrating such rows one at a time would + destroy the content. + + * A few rows decode identically with and without the dictionary: tiny, + already-compressed payloads where the compressor found nothing to reference. + Those are left alone, so "already migrated" covers them as well as genuinely + dictionary-bound rows, and re-running does not churn them. + + * The continuation rows in that database are numbered from **-2**, while + WebServer's reassembly loop starts at -1 (ADFA-5170), so those items already + serve truncated. This script preserves whatever numbering it finds, keeping + the migration behaviour-neutral; --renumber-continuations rewrites them from + -1 instead, which incidentally makes them reachable again. + +Parallel by default: compression at quality 11 is the whole cost (~4 GB of +plaintext), and it parallelises perfectly across cores. + +Usage: + # inspect: what would change, nothing written + migrate_content_to_dictionary_brotli.py documentation.db --dry-run + + # migrate a copy, then swap it in + cp documentation.db migrated.db + migrate_content_to_dictionary_brotli.py migrated.db --yes + +Requires the `brotli` CLI (>= 1.0) on PATH: no Python binding exposes custom +dictionaries, so encode and decode both shell out to it with -D. +""" + +from __future__ import annotations + +import argparse +import concurrent.futures as futures +import os +import re +import sqlite3 +import subprocess +import sys +import tempfile +import time +from dataclasses import dataclass, field + +CHUNK_BYTES = 1024 * 1024 +CONTINUATION = re.compile(r"^(.*)-(\d+)$") + +# Set once per worker process: the dictionary lives in a file because the CLI +# takes a path, and writing it once per process beats once per row. +_DICTIONARY_PATH = "" + + +def _init_worker(dictionary: bytes) -> None: + global _DICTIONARY_PATH + handle, path = tempfile.mkstemp(prefix="brotli-dict-", suffix=".bin") + with os.fdopen(handle, "wb") as out: + out.write(dictionary) + _DICTIONARY_PATH = path + + +def _brotli(args: list[str], payload: bytes) -> tuple[bool, bytes, str]: + """Run the brotli CLI over stdin/stdout. Returns (ok, output, stderr).""" + done = subprocess.run(["brotli", *args], input=payload, capture_output=True) + return done.returncode == 0, done.stdout, done.stderr.decode("utf-8", "replace").strip() + + +def decode_plain(payload: bytes) -> tuple[bool, bytes]: + ok, out, _ = _brotli(["-d", "-c"], payload) + return ok, out + + +def decode_with_dictionary(payload: bytes) -> tuple[bool, bytes]: + ok, out, _ = _brotli(["-d", "-D", _DICTIONARY_PATH, "-c"], payload) + return ok, out + + +def encode_with_dictionary(payload: bytes, quality: int, window: int) -> tuple[bool, bytes, str]: + return _brotli( + ["-q", str(quality), "-w", str(window), "-D", _DICTIONARY_PATH, "-c", "-f"], + payload, + ) + + +@dataclass +class Item: + """One logical piece of content: a base row plus any continuation rows.""" + + base_path: str + base_id: int + language_id: int + content_type_id: int + template_id: int + # (row id, suffix number, byte length), ascending by suffix + continuations: list[tuple[int, int, int]] = field(default_factory=list) + base_bytes: int = 0 + + @property + def stored_bytes(self) -> int: + return self.base_bytes + sum(n for _, _, n in self.continuations) + + @property + def first_suffix(self) -> int: + return self.continuations[0][1] if self.continuations else 1 + + +@dataclass +class Result: + base_path: str + status: str # migrated | already | unchanged | error + slices: list[bytes] = field(default_factory=list) + before: int = 0 + after: int = 0 + detail: str = "" + + +def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only_if_smaller: bool) -> Result: + """Decode an item, recompress it against the dictionary, and re-split it. + + Classification deliberately tries the *plain* decode first. Attaching no + dictionary to a stream that needs one reliably fails, so a successful plain + decode proves the row is not yet migrated; the reverse test is not safe, + because a dictionary attached to a stream that never used one can decode to + different bytes without erroring. + """ + stored = b"".join(blobs) + before = len(stored) + + ok, plaintext = decode_plain(stored) + if not ok: + ok_dict, _ = decode_with_dictionary(stored) + if ok_dict: + return Result(item.base_path, "already", before=before, after=before) + return Result(item.base_path, "error", before=before, detail="decodes neither plainly nor with the dictionary") + + # A stream that decodes *both* ways is one the compressor never referenced the + # dictionary for -- small, already-compressed payloads like a 1 KB GIF. It is + # byte-identical in either form, so there is nothing to migrate, and skipping it + # keeps a re-run from recompressing it for no gain. + ok_dict, as_dict = decode_with_dictionary(stored) + if ok_dict and as_dict == plaintext: + return Result(item.base_path, "already", before=before, after=before) + + ok, recompressed, stderr = encode_with_dictionary(plaintext, quality, window) + if not ok: + return Result(item.base_path, "error", before=before, detail=f"compression failed: {stderr}") + + # The migration is only worth anything if it round-trips exactly. + ok, roundtrip = decode_with_dictionary(recompressed) + if not ok or roundtrip != plaintext: + return Result( + item.base_path, + "error", + before=before, + detail="recompressed bytes do not decode back to the original content", + ) + + if only_if_smaller and len(recompressed) >= before: + return Result(item.base_path, "unchanged", before=before, after=before, + detail=f"dictionary-compressed form is larger ({len(recompressed)} vs {before})") + + slices = [recompressed[i:i + CHUNK_BYTES] for i in range(0, len(recompressed), CHUNK_BYTES)] or [b""] + return Result(item.base_path, "migrated", slices=slices, before=before, after=len(recompressed)) + + +def load_items(connection: sqlite3.Connection) -> list[Item]: + rows = connection.execute( + """ + SELECT C.id, C.path, C.languageID, C.contentTypeID, C.templateId, LENGTH(C.content) + FROM Content C + JOIN ContentTypes CT ON CT.id = C.contentTypeID + WHERE CT.compression = 'brotli' + """ + ).fetchall() + + by_path = {path: row for row in rows for path in (row[1],)} + items: dict[str, Item] = {} + continuations: list[tuple[str, int, int, int]] = [] + + for row_id, path, language_id, content_type_id, template_id, length in rows: + match = CONTINUATION.match(path) + # A continuation only counts as one if its base is itself a row; a path + # that merely ends in - is ordinary content. + if match and match.group(1) in by_path: + continuations.append((match.group(1), row_id, int(match.group(2)), length)) + else: + items[path] = Item(path, row_id, language_id, content_type_id, template_id, base_bytes=length) + + for base_path, row_id, suffix, length in continuations: + owner = items.get(base_path) + if owner is not None: + owner.continuations.append((row_id, suffix, length)) + + for item in items.values(): + item.continuations.sort(key=lambda entry: entry[1]) + + return sorted(items.values(), key=lambda item: item.base_path) + + +def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes]: + ids = [item.base_id] + [row_id for row_id, _, _ in item.continuations] + placeholders = ",".join("?" * len(ids)) + found = dict(connection.execute(f"SELECT id, content FROM Content WHERE id IN ({placeholders})", ids).fetchall()) + return [found[row_id] for row_id in ids] + + +def write_item(connection: sqlite3.Connection, item: Item, slices: list[bytes], renumber: bool) -> tuple[int, int]: + """Write an item's new slices back. Returns (rows inserted, rows deleted).""" + connection.execute("UPDATE Content SET content = ? WHERE id = ?", (slices[0], item.base_id)) + + start = 1 if renumber else item.first_suffix + wanted = list(enumerate(slices[1:], start=start)) + existing = {suffix: row_id for row_id, suffix, _ in item.continuations} + inserted = deleted = 0 + + for suffix, payload in wanted: + row_id = existing.pop(suffix, None) + if row_id is None: + connection.execute( + """ + INSERT INTO Content (path, languageID, content, contentTypeID, templateId) + VALUES (?, ?, ?, ?, ?) + """, + (f"{item.base_path}-{suffix}", item.language_id, payload, item.content_type_id, item.template_id), + ) + inserted += 1 + else: + connection.execute("UPDATE Content SET content = ? WHERE id = ?", (payload, row_id)) + + # Whatever is left over described slices the new stream no longer needs. + for row_id in existing.values(): + connection.execute("DELETE FROM Content WHERE id = ?", (row_id,)) + deleted += 1 + + return inserted, deleted + + +def human(n: float) -> str: + for unit in ("B", "KiB", "MiB", "GiB"): + if abs(n) < 1024 or unit == "GiB": + return f"{n:,.1f} {unit}" if unit != "B" else f"{n:,.0f} B" + n /= 1024 + return f"{n:,.1f} GiB" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("database", help="documentation.db to migrate (operate on a copy)") + parser.add_argument("--yes", action="store_true", help="actually write; without it the run is a dry run") + parser.add_argument("--dry-run", action="store_true", help="explicit no-write run (the default anyway)") + parser.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 2)), help="parallel compressors") + parser.add_argument("--quality", type=int, default=11, help="brotli quality (default 11, as the pipeline uses)") + parser.add_argument("--window", type=int, default=22, help="brotli window log (default 22, the portable maximum)") + parser.add_argument("--limit", type=int, default=0, help="stop after this many items (for a smoke test)") + parser.add_argument("--path", default="", help="only items whose base path contains this substring") + parser.add_argument("--batch", type=int, default=200, help="items per write transaction") + parser.add_argument( + "--only-if-smaller", + action="store_true", + help="leave a row alone when its dictionary-compressed form is not smaller", + ) + parser.add_argument( + "--renumber-continuations", + action="store_true", + help="write continuation rows from -1 rather than preserving existing numbering " + "(fixes ADFA-5170's unreachable slices; changes behaviour, so opt-in)", + ) + args = parser.parse_args() + write = args.yes and not args.dry_run + + connection = sqlite3.connect(args.database) + connection.execute("PRAGMA foreign_keys = ON") + + dictionary_row = connection.execute("SELECT data FROM CompressionDictionary WHERE id = 1").fetchone() + if dictionary_row is None or not dictionary_row[0]: + print("error: this database has no CompressionDictionary row to migrate against", file=sys.stderr) + return 2 + dictionary = dictionary_row[0] + + items = load_items(connection) + if args.path: + items = [item for item in items if args.path in item.base_path] + if args.limit: + items = items[: args.limit] + chunked = [item for item in items if item.continuations] + + print(f"database {args.database}") + print(f"dictionary {human(len(dictionary))}") + print(f"items {len(items):,} ({len(chunked)} of them stored as multiple slices)") + print(f"stored now {human(sum(item.stored_bytes for item in items))}") + print(f"workers {args.workers} quality {args.quality} window {args.window}") + print(f"mode {'WRITING' if write else 'dry run (pass --yes to write)'}") + if chunked and not args.renumber_continuations: + starts = sorted({item.first_suffix for item in chunked}) + print(f"continuations preserving existing numbering (starts at {starts}); " + f"--renumber-continuations rewrites from -1") + print() + + counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} + before_total = after_total = 0 + inserted_total = deleted_total = 0 + errors: list[Result] = [] + started = time.time() + + with futures.ProcessPoolExecutor(args.workers, initializer=_init_worker, initargs=(dictionary,)) as pool: + for offset in range(0, len(items), args.batch): + batch = items[offset : offset + args.batch] + pending = { + pool.submit( + migrate_item, item, read_blobs(connection, item), args.quality, args.window, args.only_if_smaller + ): item + for item in batch + } + + for future in futures.as_completed(pending): + item = pending[future] + result = future.result() + counts[result.status] += 1 + before_total += result.before + after_total += result.after or result.before + + if result.status == "error": + errors.append(result) + elif result.status == "migrated" and write: + inserted, deleted = write_item(connection, item, result.slices, args.renumber_continuations) + inserted_total += inserted + deleted_total += deleted + + if write: + connection.commit() + + done = min(offset + args.batch, len(items)) + elapsed = time.time() - started + rate = done / elapsed if elapsed else 0 + remaining = (len(items) - done) / rate if rate else 0 + print( + f"\r{done:,}/{len(items):,} items {rate:5.1f}/s " + f"eta {remaining/60:4.1f} min saved {human(before_total - after_total)}", + end="", + flush=True, + ) + + print("\n") + print(f"migrated {counts['migrated']:,}") + print(f"already {counts['already']:,}") + if counts["unchanged"]: + print(f"left alone {counts['unchanged']:,} (not smaller with the dictionary)") + print(f"errors {counts['error']:,}") + if write: + print(f"rows inserted {inserted_total} rows deleted {deleted_total}") + print(f"stored before {human(before_total)}") + print(f"stored after {human(after_total)}") + if before_total: + print(f"saved {human(before_total - after_total)} ({100 * (before_total - after_total) / before_total:.1f}%)") + print(f"took {(time.time() - started)/60:.1f} min") + + for result in errors[:20]: + print(f" error: {result.base_path}: {result.detail}", file=sys.stderr) + if len(errors) > 20: + print(f" ... and {len(errors) - 20} more", file=sys.stderr) + + if write: + connection.commit() + print("\nRun VACUUM to reclaim the freed pages: sqlite3 %s 'VACUUM;'" % args.database) + else: + connection.rollback() + print("\nNothing written. Re-run with --yes on a copy to apply.") + + connection.close() + return 1 if errors else 0 + + +if __name__ == "__main__": + sys.exit(main()) From 2ea1ba86639411db4e8b747ebc67332bf54625ab Mon Sep 17 00:00:00 2001 From: David Schachter Date: Thu, 20 Aug 2026 16:29:39 -0700 Subject: [PATCH 02/16] ADFA-5153: Keep Spotless's shell rules off Python scripts The `shell` block targets scripts/** wholesale and runs leadingSpacesToTabs(), so adding a .py file there gets it reindented to tabs -- against PEP 8, and against every .py already in this repo, all of which are space-indented. Only the ratchet has been hiding that: those files never differ from origin/stage, so Spotless never touches them. The first edit to scripts/cloudflare-r2-upload.py or scripts/insert-ci-perf-data.py would have silently converted the whole file, which is a trap worth removing rather than working around. --- build.gradle.kts | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/build.gradle.kts b/build.gradle.kts index b0f6438ace..9c3dd7ee92 100755 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -332,7 +332,13 @@ spotless { ".githooks/**/*", "scripts/**/*", ) - targetExclude("scripts/debug-keystore/adfa-keystore.jks") + targetExclude( + "scripts/debug-keystore/adfa-keystore.jks", + // leadingSpacesToTabs() would reindent Python, which PEP 8 indents with spaces -- + // and every .py already here is space-indented. Only the ratchet has been hiding + // that mismatch: an edit to one of them would silently convert the whole file. + "**/*.py", + ) } } From 5ef824040a88b47170ac77fd287fb4edd43e4b78 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Thu, 20 Aug 2026 17:02:11 -0700 Subject: [PATCH 03/16] ADFA-5153: Repair mislabelled and mis-chunked rows before recompressing The dictionary migration now runs in three phases, because each changes what the next one sees: retype -- 74 rows hold GIF/PNG/JPEG/QuickTime payloads but are typed text/plain (ADFA-5221), so they are Brotli-compressed for no gain and served as Content-Type: text/plain. Store their plaintext and point them at the type their magic bytes prove they are. renumber -- 14 of 19 chunked items number continuations from -2 while the app's reassembly loop starts at -1 (ADFA-5170), so they serve as their first 1 MiB and nothing more. Shift them down. migrate -- the existing recompression pass, unchanged. Phase 1 feeds phase 3 for free: a row retyped to image/gif inherits that type's compression = 'none', so the compression = 'brotli' selection stops seeing it. No exclusion list needed. Extensions only nominate phase 1's candidates; magic bytes decide, and a name/content disagreement is reported rather than trusted. The four .mov files are ftypqt QuickTime, not ISO-BMFF, so --mov-type chooses between the honest video/quicktime (inserted into ContentTypes as id 28) and the video/mp4 Chromium is likelier to play. Verified on a copy of the 20-Aug database: 74/74 retyped rows byte-identical to the original plaintext, all 19 chunked items reassembling to unchanged bytes, 250/250 sampled rows decoding with the dictionary to identical content, integrity_check ok, no foreign-key violations, Content and Bookshelf row counts unchanged, and a second run reporting nothing left to do. 3.5 min at 20 workers; 313.8 -> 268.1 MB after VACUUM. Co-Authored-By: Claude Opus 5 (1M context) --- docs/documentation-database.md | 3 +- .../migrate_content_to_dictionary_brotli.py | 511 +++++++++++++++--- 2 files changed, 451 insertions(+), 63 deletions(-) diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 355861fdd2..75674d6e2b 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -36,6 +36,7 @@ One row per file the web server can serve (HTML, CSS, JS, image, video, PDF, ... - **`path`** is the lookup key (indexed via the `UNIQUE` constraint) and is what `WebServer` matches the HTTP request path against. Paths carry a short source prefix to avoid collisions between doc sets, e.g. `k/index.html` (Kotlin) vs `j/index.html` (Java). - **`content`** is compressed — Brotli for text-like formats, format-specific compression otherwise (images/video/fonts). `ContentTypes.compression` says which. Every migrated `Content` row with `ContentTypes.compression = 'brotli'` is Brotli-compressed against the single shared dictionary in `CompressionDictionary` (see below), converted in one pass by ADFA-5153 — but plugin-contributed Tier 3 rows (`PluginDocumentationManager`/`BrotliCompressor`, see below) are plain, dictionary-free Brotli, and there is no per-row flag distinguishing the two, because a dictionary-compressed stream and a plain one are not distinguishable at decode time by inspection. They *are* distinguishable by attempting the decode: attaching the *wrong* dictionary decodes without error to different bytes than were compressed (its backward distances resolve into real, just incorrect, bytes) — but attaching *no* dictionary to a stream that needs one reliably throws (`IOException`, "corrupted input"), since distances into the dictionary region are then out of bounds for any spec-compliant decoder. `WebServer` relies on exactly this: it tries the dictionary first and falls back to a plain decode on `IOException`, which correctly handles both dictionary-compressed and plain rows — but never rely on decode success/failure to detect a *wrong* dictionary, since that case is silent. Content over 1 MB is split across multiple rows: the first row's path is the base path, continuation rows are `path-1`, `path-2`, ... (`languageId = 1`), reassembled by `WebServer` before returning. - **`templateId`**: `0` (or unset) means `content` is legacy HTML with presentation baked in (the pre-CMS Release 0/1 format). A positive value means `content` is JSON *facts only*, rendered through the matching row in `Templates` (a Pebble template) — the ongoing move to a proper CMS that de-duplicates presentation across near-identical pages (e.g. `sin`/`cos` docs). +- Two data defects live in the shipped rows rather than in the schema, and `scripts/docdb/migrate_content_to_dictionary_brotli.py` repairs both before it recompresses anything. **Chunk numbering:** 14 of the 19 chunked items number their continuations from `-2`, not the `-1` the reassembly loop starts at (ADFA-5170), so those items serve as their first 1 MiB and nothing more; the script's `renumber` phase shifts them down. **Mislabelled types:** 74 rows holding GIF/PNG/JPEG/QuickTime payloads are typed `text/plain` (ADFA-5221), so they are Brotli-compressed for no gain and served as `Content-Type: text/plain`; the `retype` phase stores their plaintext and points them at the type their magic bytes prove they are, which -- since those types carry `compression = 'none'` -- also drops them out of the dictionary pass. Both defects originate in `docdb-studio`'s import path, so a freshly exported database will carry them again until fixed there. - The `UNIQUE(path)` constraint rejects any duplicate `path`, regardless of `languageID` — a second language for an existing path isn't supported yet (only `EN-us` currently exists). Getting there needs an upstream schema change to composite uniqueness on `(path, languageID)` (see *Known rough edges* below). Dimensions: `Languages(id, value)` (4-letter codes, e.g. `EN-us`); `ContentTypes(id, value, compression)` (MIME type + compression scheme, ~30 rows). @@ -63,7 +64,7 @@ CREATE TABLE Tooltips ( ### Supporting tables - **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. Append-only: each change is another `INSERT`, so the **row inserted last** is the current version, not the highest one ever recorded — a rebuild from an older content set is a downgrade and has to read as one (`DatabaseVersionResolver.resolveMajorVersion`, which returns null for a database predating the table). `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `WebServer` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every row would then pay a failed dictionary decode before its plain one, on every request), and a migrated database that lost the table fails loudly instead of quietly. -- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows)(never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). +- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows, after repairing the two data defects noted above)(never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). - **`Templates(id, name, content)`** — Pebble template source, keyed by id (and by `name` for well-known templates like `bookshelf`). Referenced by `Content.templateId`. - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`). - **`LastChange(documentationSet, changeTime, who)`** — audit trail for edits made through `docdb-studio`; not shown to end users. `DatabaseVersionResolver` reads the `documentationSet = 'wholedb'` row to report the DB's build/edit stamp in debug logging, falling back to the most recent row of any set if `'wholedb'` is missing. diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index af0f6dc885..7a592d0703 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -1,43 +1,68 @@ #!/usr/bin/env python3 -"""Recompress documentation.db's Brotli Content rows against the shared dictionary. - -Reads the dictionary from the database's own CompressionDictionary table (id = 1) -and rewrites every `ContentTypes.compression = 'brotli'` row so it is compressed -against that dictionary instead of plainly. WebServer tries a dictionary-attached -decode first and falls back to a plain one, so a half-migrated database still -serves -- which is what makes running this incrementally safe. - -Two properties of the data shape this script, both verified against the 20-Aug +"""Repair mislabelled binary rows in documentation.db, then recompress its Brotli +Content rows against the shared dictionary. + +Three phases, in this order, because each one changes what the next one sees: + + 1. retype -- rows that claim to be text but hold a GIF/PNG/JPEG/QuickTime + payload (ADFA-5221). Their declared type is `text/plain`, whose + ContentTypes row says `brotli`, so they were pointlessly + compressed *and* are served as `Content-Type: text/plain`. The + fix is to store the plaintext and point the row at the honest + type, whose compression is `none`. + 2. renumber -- chunked items whose continuation rows start at -2 while + WebServer's reassembly loop starts at -1 (ADFA-5170), so they + currently serve as their first 1 MiB and nothing more. + 3. migrate -- rewrite every `ContentTypes.compression = 'brotli'` row so it is + compressed against the database's own dictionary rather than + plainly. WebServer tries a dictionary-attached decode first and + falls back to a plain one, so a half-migrated database still + serves -- which is what makes running this incrementally safe. + +Phase 1 feeds phase 3 for free: a row retyped to `image/gif` inherits that type's +`compression = 'none'`, so phase 3's `compression = 'brotli'` selection simply +stops seeing it. No exclusion list is needed. + +Properties of the data that shape this script, all verified against the 20-Aug database rather than assumed: * Content over 1 MiB is *not* stored as independently compressed pieces. The - rows are raw 1 MiB slices of one Brotli stream: `path`, then `path-N` - continuations. A slice on its own does not decode. So the unit of work here - is a logical item -- a base row plus its continuations -- concatenated, - decoded, recompressed, and re-split. Migrating such rows one at a time would - destroy the content. + rows are raw 1 MiB slices of one stream: `path`, then `path-N` continuations. + A slice on its own does not decode. So the unit of work is a logical item -- + a base row plus its continuations -- concatenated, decoded, rewritten, and + re-split. Treating such rows one at a time would destroy the content. * A few rows decode identically with and without the dictionary: tiny, already-compressed payloads where the compressor found nothing to reference. Those are left alone, so "already migrated" covers them as well as genuinely dictionary-bound rows, and re-running does not churn them. - * The continuation rows in that database are numbered from **-2**, while - WebServer's reassembly loop starts at -1 (ADFA-5170), so those items already - serve truncated. This script preserves whatever numbering it finds, keeping - the migration behaviour-neutral; --renumber-continuations rewrites them from - -1 instead, which incidentally makes them reachable again. + * Extensions nominate phase 1's candidates; magic bytes decide. A row is + retyped to what its payload actually is, not to what its name suggests, and a + name/content disagreement is reported rather than trusted. The four `.mov` + files are `ftypqt` QuickTime, not ISO-BMFF, so --mov-type picks between the + honest `video/quicktime` (inserted into ContentTypes if absent) and the + `video/mp4` that Chromium is likelier to actually play. + + * `Content` has a real UNIQUE constraint on `path` (the schema's `UNIQUE('path')` + quotes the identifier but does enforce it), so a renumbering mistake fails + loudly instead of duplicating a row. The `AddBook`/`DeleteBook` triggers fire + only for paths ending `.pdf`, which continuation paths never do. Parallel by default: compression at quality 11 is the whole cost (~4 GB of plaintext), and it parallelises perfectly across cores. Usage: - # inspect: what would change, nothing written + # inspect: what all three phases would change, nothing written migrate_content_to_dictionary_brotli.py documentation.db --dry-run - # migrate a copy, then swap it in + # do it, on a copy cp documentation.db migrated.db migrate_content_to_dictionary_brotli.py migrated.db --yes + sqlite3 migrated.db 'VACUUM;' + + # just the data repair, leaving compression alone + migrate_content_to_dictionary_brotli.py migrated.db --yes --phases retype,renumber Requires the `brotli` CLI (>= 1.0) on PATH: no Python binding exposes custom dictionaries, so encode and decode both shell out to it with -D. @@ -58,6 +83,15 @@ CHUNK_BYTES = 1024 * 1024 CONTINUATION = re.compile(r"^(.*)-(\d+)$") +ALL_PHASES = ("retype", "renumber", "migrate") + +# Extensions worth a second look when a row claims to be text. The extension only +# nominates a candidate -- sniff() decides what the row actually holds. +BINARY_EXTENSIONS = ( + ".gif", ".png", ".jpg", ".jpeg", ".webp", ".ico", ".bmp", + ".mov", ".mp4", ".m4v", ".pdf", + ".woff", ".woff2", ".ttf", ".otf", ".wasm", +) # Set once per worker process: the dictionary lives in a file because the CLI # takes a path, and writing it once per process beats once per row. @@ -95,6 +129,40 @@ def encode_with_dictionary(payload: bytes, quality: int, window: int) -> tuple[b ) +def sniff(payload: bytes) -> str: + """The MIME type the bytes themselves declare, or '' if unrecognised.""" + if payload[:6] in (b"GIF87a", b"GIF89a"): + return "image/gif" + if payload[:8] == b"\x89PNG\r\n\x1a\n": + return "image/png" + if payload[:3] == b"\xff\xd8\xff": + return "image/jpeg" + if payload[:4] == b"RIFF" and payload[8:12] == b"WEBP": + return "image/webp" + if payload[:4] == b"\x00\x00\x01\x00": + return "image/x-icon" + if payload[:4] == b"%PDF": + return "application/pdf" + if payload[4:8] == b"ftyp": + # The brand distinguishes a QuickTime container from ISO-BMFF/MP4. + return "video/quicktime" if payload[8:12] == b"qt " else "video/mp4" + if payload[:4] == b"wOF2": + return "font/woff2" + if payload[:4] == b"wOFF": + return "font/woff" + if payload[:4] == b"OTTO": + return "font/otf" + if payload[:4] in (b"\x00\x01\x00\x00", b"true", b"ttcf"): + return "font/ttf" + if payload[:4] == b"\x00asm": + return "application/wasm" + return "" + + +def slice_stream(payload: bytes) -> list[bytes]: + return [payload[i:i + CHUNK_BYTES] for i in range(0, len(payload), CHUNK_BYTES)] or [b""] + + @dataclass class Item: """One logical piece of content: a base row plus any continuation rows.""" @@ -104,6 +172,8 @@ class Item: language_id: int content_type_id: int template_id: int + content_type: str = "" + compression: str = "" # (row id, suffix number, byte length), ascending by suffix continuations: list[tuple[int, int, int]] = field(default_factory=list) base_bytes: int = 0 @@ -116,15 +186,59 @@ def stored_bytes(self) -> int: def first_suffix(self) -> int: return self.continuations[0][1] if self.continuations else 1 + @property + def suffixes(self) -> list[int]: + return [suffix for _, suffix, _ in self.continuations] + + +@dataclass +class Inspection: + """Phase 1's verdict on one candidate row.""" + + base_path: str + status: str # retype | keep | error + sniffed: str = "" + slices: list[bytes] = field(default_factory=list) + before: int = 0 + after: int = 0 + detail: str = "" + @dataclass class Result: + """Phase 3's verdict on one item.""" + base_path: str status: str # migrated | already | unchanged | error slices: list[bytes] = field(default_factory=list) before: int = 0 after: int = 0 detail: str = "" + sniffed: str = "" # set when a text-typed row turns out to hold binary + + +def inspect_item(item: Item, blobs: list[bytes]) -> Inspection: + """Decide what a text-typed candidate actually holds, and hand back its plaintext.""" + stored = b"".join(blobs) + before = len(stored) + + if item.compression == "brotli": + ok, payload = decode_plain(stored) + if not ok: + ok, payload = decode_with_dictionary(stored) + if not ok: + return Inspection(item.base_path, "error", before=before, + detail="decodes neither plainly nor with the dictionary") + else: + payload = stored + + kind = sniff(payload) + if not kind: + return Inspection(item.base_path, "keep", before=before, after=before, + detail=f"declared {item.content_type}, and the payload carries no binary signature") + + return Inspection(item.base_path, "retype", sniffed=kind, slices=slice_stream(payload), + before=before, after=len(payload)) def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only_if_smaller: bool) -> Result: @@ -146,13 +260,17 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only return Result(item.base_path, "already", before=before, after=before) return Result(item.base_path, "error", before=before, detail="decodes neither plainly nor with the dictionary") + # Decoding every row here anyway makes a mislabel sweep free: report a + # text-typed row whose payload is recognisably binary, whatever its name. + mislabelled = sniff(plaintext) if item.content_type.startswith("text") else "" + # A stream that decodes *both* ways is one the compressor never referenced the # dictionary for -- small, already-compressed payloads like a 1 KB GIF. It is # byte-identical in either form, so there is nothing to migrate, and skipping it # keeps a re-run from recompressing it for no gain. ok_dict, as_dict = decode_with_dictionary(stored) if ok_dict and as_dict == plaintext: - return Result(item.base_path, "already", before=before, after=before) + return Result(item.base_path, "already", before=before, after=before, sniffed=mislabelled) ok, recompressed, stderr = encode_with_dictionary(plaintext, quality, window) if not ok: @@ -169,35 +287,45 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only ) if only_if_smaller and len(recompressed) >= before: - return Result(item.base_path, "unchanged", before=before, after=before, + return Result(item.base_path, "unchanged", before=before, after=before, sniffed=mislabelled, detail=f"dictionary-compressed form is larger ({len(recompressed)} vs {before})") - slices = [recompressed[i:i + CHUNK_BYTES] for i in range(0, len(recompressed), CHUNK_BYTES)] or [b""] - return Result(item.base_path, "migrated", slices=slices, before=before, after=len(recompressed)) + return Result(item.base_path, "migrated", slices=slice_stream(recompressed), + before=before, after=len(recompressed), sniffed=mislabelled) -def load_items(connection: sqlite3.Connection) -> list[Item]: +def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: rows = connection.execute( - """ - SELECT C.id, C.path, C.languageID, C.contentTypeID, C.templateId, LENGTH(C.content) + f""" + SELECT C.id, C.path, C.languageID, C.contentTypeID, C.templateId, + LENGTH(C.content), CT.value, CT.compression FROM Content C JOIN ContentTypes CT ON CT.id = C.contentTypeID - WHERE CT.compression = 'brotli' + WHERE {predicate} """ ).fetchall() - by_path = {path: row for row in rows for path in (row[1],)} + paths = {row[1] for row in rows} items: dict[str, Item] = {} continuations: list[tuple[str, int, int, int]] = [] - for row_id, path, language_id, content_type_id, template_id, length in rows: + for row_id, path, language_id, type_id, template_id, length, type_value, compression in rows: match = CONTINUATION.match(path) # A continuation only counts as one if its base is itself a row; a path # that merely ends in - is ordinary content. - if match and match.group(1) in by_path: + if match and match.group(1) in paths: continuations.append((match.group(1), row_id, int(match.group(2)), length)) else: - items[path] = Item(path, row_id, language_id, content_type_id, template_id, base_bytes=length) + items[path] = Item( + base_path=path, + base_id=row_id, + language_id=language_id, + content_type_id=type_id, + template_id=template_id, + content_type=type_value, + compression=compression, + base_bytes=length, + ) for base_path, row_id, suffix, length in continuations: owner = items.get(base_path) @@ -217,15 +345,35 @@ def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes]: return [found[row_id] for row_id in ids] -def write_item(connection: sqlite3.Connection, item: Item, slices: list[bytes], renumber: bool) -> tuple[int, int]: +def write_item( + connection: sqlite3.Connection, + item: Item, + slices: list[bytes], + renumber: bool, + content_type_id: int | None = None, +) -> tuple[int, int]: """Write an item's new slices back. Returns (rows inserted, rows deleted).""" - connection.execute("UPDATE Content SET content = ? WHERE id = ?", (slices[0], item.base_id)) + type_id = item.content_type_id if content_type_id is None else content_type_id + connection.execute( + "UPDATE Content SET content = ?, contentTypeID = ? WHERE id = ?", + (slices[0], type_id, item.base_id), + ) start = 1 if renumber else item.first_suffix wanted = list(enumerate(slices[1:], start=start)) existing = {suffix: row_id for row_id, suffix, _ in item.continuations} inserted = deleted = 0 + # Retained rows are renumbered by taking the slot they now hold, so drop every + # old continuation path first and re-create what the new stream needs. Deleting + # before inserting keeps the UNIQUE(path) constraint out of the way when the + # numbering shifts. + if renumber and item.first_suffix != 1: + for row_id in existing.values(): + connection.execute("DELETE FROM Content WHERE id = ?", (row_id,)) + deleted += 1 + existing = {} + for suffix, payload in wanted: row_id = existing.pop(suffix, None) if row_id is None: @@ -234,11 +382,14 @@ def write_item(connection: sqlite3.Connection, item: Item, slices: list[bytes], INSERT INTO Content (path, languageID, content, contentTypeID, templateId) VALUES (?, ?, ?, ?, ?) """, - (f"{item.base_path}-{suffix}", item.language_id, payload, item.content_type_id, item.template_id), + (f"{item.base_path}-{suffix}", item.language_id, payload, type_id, item.template_id), ) inserted += 1 else: - connection.execute("UPDATE Content SET content = ? WHERE id = ?", (payload, row_id)) + connection.execute( + "UPDATE Content SET content = ?, contentTypeID = ? WHERE id = ?", + (payload, type_id, row_id), + ) # Whatever is left over described slices the new stream no longer needs. for row_id in existing.values(): @@ -248,6 +399,59 @@ def write_item(connection: sqlite3.Connection, item: Item, slices: list[bytes], return inserted, deleted +def retype_rows(connection: sqlite3.Connection, item: Item, content_type_id: int) -> None: + """Point an item's rows at a different content type, leaving the bytes alone.""" + ids = [item.base_id] + [row_id for row_id, _, _ in item.continuations] + placeholders = ",".join("?" * len(ids)) + connection.execute( + f"UPDATE Content SET contentTypeID = ? WHERE id IN ({placeholders})", + [content_type_id, *ids], + ) + + +def content_types(connection: sqlite3.Connection) -> dict[str, tuple[int, str]]: + return { + value: (type_id, compression) + for type_id, value, compression in connection.execute("SELECT id, value, compression FROM ContentTypes") + } + + +def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> str: + """Shift an item's continuations down so they start at -1. Returns a note, or ''.""" + shift = item.first_suffix - 1 + if shift <= 0: + return "" + + expected = list(range(item.first_suffix, item.first_suffix + len(item.continuations))) + if item.suffixes != expected: + return f"continuations are not contiguous ({item.suffixes}); left alone" + + # A row this item already owns is not a clash: it is the one being moved out of + # that slot. Only a foreign row occupying a target path blocks the shift. + own = {row_id for row_id, _, _ in item.continuations} + for _, suffix, _ in item.continuations: + target = f"{item.base_path}-{suffix - shift}" + clash = connection.execute("SELECT id FROM Content WHERE path = ?", (target,)).fetchone() + if clash and clash[0] not in own: + return f"{target} already exists and belongs to another row; left alone" + + if write: + # Two passes: park every row under a name nothing can collide with, then + # settle them into their new slots. One ascending pass would be enough only + # if the target were always already free, and for a shift of 1 it never is. + for row_id, suffix, _ in item.continuations: + connection.execute( + "UPDATE Content SET path = ? WHERE id = ?", + (f"{item.base_path}-renumbering-{suffix}", row_id), + ) + for row_id, suffix, _ in item.continuations: + connection.execute( + "UPDATE Content SET path = ? WHERE id = ?", + (f"{item.base_path}-{suffix - shift}", row_id), + ) + return "" + + def human(n: float) -> str: for unit in ("B", "KiB", "MiB", "GiB"): if abs(n) < 1024 or unit == "GiB": @@ -256,11 +460,152 @@ def human(n: float) -> str: return f"{n:,.1f} GiB" +def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[str]]: + """Retype rows that claim to be text but hold a recognisable binary payload.""" + print(f"[1/3] retype {len(items):,} text-typed rows with a binary extension") + if not items: + return set(), [] + + types = content_types(connection) + counts: dict[str, int] = {} + problems: list[str] = [] + retyped: list[tuple[Item, Inspection, str]] = [] + before_total = after_total = 0 + + pending = {pool.submit(inspect_item, item, read_blobs(connection, item)): item for item in items} + for future in futures.as_completed(pending): + item = pending[future] + found = future.result() + if found.status == "error": + problems.append(f"{item.base_path}: {found.detail}") + continue + if found.status == "keep": + counts["left as text"] = counts.get("left as text", 0) + 1 + problems.append(f"{item.base_path}: {found.detail}") + continue + + target = found.sniffed + if target == "video/quicktime" and args.mov_type == "mp4": + target = "video/mp4" + extension = item.base_path.rsplit(".", 1)[-1].lower() + if extension not in target and not (extension in ("jpg", "jpeg") and target == "image/jpeg") \ + and not (extension == "mov" and target.startswith("video/")): + problems.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") + + retyped.append((item, found, target)) + counts[target] = counts.get(target, 0) + 1 + before_total += found.before + after_total += found.after + + missing = sorted({target for _, _, target in retyped if target not in types}) + for value in missing: + if write: + cursor = connection.execute( + "INSERT INTO ContentTypes (value, compression) VALUES (?, 'none')", (value,) + ) + types[value] = (cursor.lastrowid, "none") + print(f" ContentTypes + id {cursor.lastrowid} {value} (compression none)") + else: + types[value] = (-1, "none") + print(f" ContentTypes would insert {value} (compression none)") + + inserted_total = deleted_total = 0 + for item, found, target in retyped: + type_id, compression = types[target] + if not write: + continue + if compression == "none": + inserted, deleted = write_item( + connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id + ) + inserted_total += inserted + deleted_total += deleted + else: + # The honest type is itself a compressed one, so the stored bytes stay + # as they are and phase 3 picks the row up. + retype_rows(connection, item, type_id) + if write: + connection.commit() + + for target, count in sorted(counts.items(), key=lambda kv: -kv[1]): + print(f" {target:22} {count:>4}") + print(f" stored {human(before_total)} of Brotli -> {human(after_total)} of plaintext " + f"({'+' if after_total >= before_total else ''}{human(after_total - before_total)})") + if write: + print(f" rows inserted {inserted_total} deleted {deleted_total}") + return {item.base_path for item, _, _ in retyped}, problems + + +def phase_renumber(connection, args, write, retyped_paths: set[str]) -> tuple[int, list[str]]: + """Shift -2-based continuation numbering down to the -1 the app expects.""" + items = [item for item in load_items(connection, "1 = 1") if item.continuations] + if args.renumber_scope == "retyped": + items = [item for item in items if item.base_path in retyped_paths] + broken = [item for item in items if item.first_suffix != 1] + + starts = sorted({item.first_suffix for item in broken}) + if broken: + print(f"[2/3] renumber {len(broken)} of {len(items)} chunked items start at " + f"{', '.join('-' + str(n) for n in starts)} instead of -1") + else: + print(f"[2/3] renumber all {len(items)} chunked items already start at -1") + problems: list[str] = [] + fixed = 0 + for item in broken: + note = renumber_item(connection, item, write) + if note: + problems.append(f"{item.base_path}: {note}") + else: + fixed += 1 + if write: + connection.commit() + + for item in items: + if item.base_bytes != CHUNK_BYTES: + problems.append( + f"{item.base_path}: chunked but its base row is {item.base_bytes:,} bytes, not " + f"{CHUNK_BYTES:,} -- the app detects chunking by that exact length, so it will not reassemble" + ) + if fixed: + print(f" renumbered from -1: {fixed}") + return fixed, problems + + +def verify_retype(connection, retyped_paths: set[str], mov_type: str) -> list[str]: + """Re-read what phase 1 wrote and confirm the bytes match the declared type.""" + problems: list[str] = [] + written = {item.base_path: item for item in load_items(connection, "1 = 1")} + for path in sorted(retyped_paths): + item = written.get(path) + if item is None: + problems.append(f"{path}: row vanished") + continue + payload = b"".join(read_blobs(connection, item)) + found = sniff(payload) + expected = item.content_type + if expected == "video/mp4" and mov_type == "mp4" and found == "video/quicktime": + found = "video/mp4" # deliberately typed mp4; the container really is qt + if found != expected: + problems.append(f"{path}: declared {expected} but the stored bytes sniff as {found or 'unknown'}") + if item.compression != "none": + problems.append(f"{path}: retyped to {expected}, whose compression is {item.compression}") + if item.continuations and item.suffixes != list(range(1, len(item.continuations) + 1)): + problems.append(f"{path}: continuations numbered {item.suffixes}, expected 1..n") + return problems + + def main() -> int: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument("database", help="documentation.db to migrate (operate on a copy)") + parser.add_argument("database", help="documentation.db to work on (operate on a copy)") parser.add_argument("--yes", action="store_true", help="actually write; without it the run is a dry run") parser.add_argument("--dry-run", action="store_true", help="explicit no-write run (the default anyway)") + parser.add_argument("--phases", default=",".join(ALL_PHASES), + help=f"comma-separated subset of {','.join(ALL_PHASES)} (default: all, in that order)") + parser.add_argument("--mov-type", choices=("quicktime", "mp4"), default="quicktime", + help="what to call the ftypqt .mov payloads: the honest video/quicktime (inserted into " + "ContentTypes) or the video/mp4 Chromium is likelier to play (default: quicktime)") + parser.add_argument("--renumber-scope", choices=("all", "retyped"), default="all", + help="renumber every -2-based chunked item, or only the ones phase 1 retyped (default: all)") parser.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 2)), help="parallel compressors") parser.add_argument("--quality", type=int, default=11, help="brotli quality (default 11, as the pipeline uses)") parser.add_argument("--window", type=int, default=22, help="brotli window log (default 22, the portable maximum)") @@ -272,15 +617,15 @@ def main() -> int: action="store_true", help="leave a row alone when its dictionary-compressed form is not smaller", ) - parser.add_argument( - "--renumber-continuations", - action="store_true", - help="write continuation rows from -1 rather than preserving existing numbering " - "(fixes ADFA-5170's unreachable slices; changes behaviour, so opt-in)", - ) args = parser.parse_args() write = args.yes and not args.dry_run + args.phase_list = [phase.strip() for phase in args.phases.split(",") if phase.strip()] + unknown = [phase for phase in args.phase_list if phase not in ALL_PHASES] + if unknown: + print(f"error: unknown phase(s) {', '.join(unknown)}; pick from {', '.join(ALL_PHASES)}", file=sys.stderr) + return 2 + connection = sqlite3.connect(args.database) connection.execute("PRAGMA foreign_keys = ON") @@ -290,32 +635,60 @@ def main() -> int: return 2 dictionary = dictionary_row[0] - items = load_items(connection) - if args.path: - items = [item for item in items if args.path in item.base_path] - if args.limit: - items = items[: args.limit] - chunked = [item for item in items if item.continuations] + def select(items: list[Item]) -> list[Item]: + if args.path: + items = [item for item in items if args.path in item.base_path] + return items[: args.limit] if args.limit else items print(f"database {args.database}") print(f"dictionary {human(len(dictionary))}") - print(f"items {len(items):,} ({len(chunked)} of them stored as multiple slices)") - print(f"stored now {human(sum(item.stored_bytes for item in items))}") + print(f"phases {' -> '.join(args.phase_list)}") print(f"workers {args.workers} quality {args.quality} window {args.window}") print(f"mode {'WRITING' if write else 'dry run (pass --yes to write)'}") - if chunked and not args.renumber_continuations: - starts = sorted({item.first_suffix for item in chunked}) - print(f"continuations preserving existing numbering (starts at {starts}); " - f"--renumber-continuations rewrites from -1") print() - counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} - before_total = after_total = 0 - inserted_total = deleted_total = 0 - errors: list[Result] = [] + problems: list[str] = [] + retyped_paths: set[str] = set() started = time.time() with futures.ProcessPoolExecutor(args.workers, initializer=_init_worker, initargs=(dictionary,)) as pool: + if "retype" in args.phase_list: + candidates = select([ + item for item in load_items(connection, "CT.value LIKE 'text%'") + if item.base_path.lower().endswith(BINARY_EXTENSIONS) + ]) + retyped_paths, notes = phase_retype(connection, pool, args, write, candidates) + problems += notes + if write: + problems += verify_retype(connection, retyped_paths, args.mov_type) + print() + + if "renumber" in args.phase_list: + _, notes = phase_renumber(connection, args, write, retyped_paths) + problems += notes + print() + + if "migrate" not in args.phase_list: + connection.commit() if write else connection.rollback() + connection.close() + sys.stdout.flush() + for note in problems[:30]: + print(f" note: {note}", file=sys.stderr) + return 0 + + items = select(load_items(connection, "CT.compression = 'brotli'")) + chunked = [item for item in items if item.continuations] + + print(f"[3/3] migrate {len(items):,} items ({len(chunked)} of them stored as multiple slices)") + print(f" stored now {human(sum(item.stored_bytes for item in items))}") + print() + + counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} + before_total = after_total = 0 + inserted_total = deleted_total = 0 + errors: list[Result] = [] + mislabelled: list[Result] = [] + for offset in range(0, len(items), args.batch): batch = items[offset : offset + args.batch] pending = { @@ -332,10 +705,12 @@ def main() -> int: before_total += result.before after_total += result.after or result.before + if result.sniffed: + mislabelled.append(result) if result.status == "error": errors.append(result) elif result.status == "migrated" and write: - inserted, deleted = write_item(connection, item, result.slices, args.renumber_continuations) + inserted, deleted = write_item(connection, item, result.slices, renumber=False) inserted_total += inserted deleted_total += deleted @@ -347,7 +722,7 @@ def main() -> int: rate = done / elapsed if elapsed else 0 remaining = (len(items) - done) / rate if rate else 0 print( - f"\r{done:,}/{len(items):,} items {rate:5.1f}/s " + f"\r {done:,}/{len(items):,} items {rate:5.1f}/s " f"eta {remaining/60:4.1f} min saved {human(before_total - after_total)}", end="", flush=True, @@ -367,10 +742,22 @@ def main() -> int: print(f"saved {human(before_total - after_total)} ({100 * (before_total - after_total) / before_total:.1f}%)") print(f"took {(time.time() - started)/60:.1f} min") + if mislabelled: + print(f"\nstill text-typed but holding binary ({len(mislabelled)}; phase 1 nominates by extension only):") + for result in mislabelled[:20]: + print(f" {result.base_path}: {result.sniffed}") + if len(mislabelled) > 20: + print(f" ... and {len(mislabelled) - 20} more") + for result in errors[:20]: print(f" error: {result.base_path}: {result.detail}", file=sys.stderr) if len(errors) > 20: print(f" ... and {len(errors) - 20} more", file=sys.stderr) + sys.stdout.flush() + for note in problems[:30]: + print(f" note: {note}", file=sys.stderr) + if len(problems) > 30: + print(f" ... and {len(problems) - 30} more notes", file=sys.stderr) if write: connection.commit() From 81147a54f18e75d5aa07b8ebbd5d975d36813295 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Fri, 21 Aug 2026 15:57:59 -0700 Subject: [PATCH 04/16] ADFA-5153: Cite ADFA-5171 for the chunk-numbering defect, not ADFA-5170 ADFA-5171 is "Chunked Content rows numbered from -2 break reassembly"; ADFA-5170 is a separate task about peak heap when serving chunked rows. The docstring and the doc paragraph both pointed at the wrong one. Co-Authored-By: Claude Opus 5 (1M context) --- docs/documentation-database.md | 4 ++-- scripts/docdb/migrate_content_to_dictionary_brotli.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/documentation-database.md b/docs/documentation-database.md index 75674d6e2b..d93f6bc9a5 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -36,7 +36,7 @@ One row per file the web server can serve (HTML, CSS, JS, image, video, PDF, ... - **`path`** is the lookup key (indexed via the `UNIQUE` constraint) and is what `WebServer` matches the HTTP request path against. Paths carry a short source prefix to avoid collisions between doc sets, e.g. `k/index.html` (Kotlin) vs `j/index.html` (Java). - **`content`** is compressed — Brotli for text-like formats, format-specific compression otherwise (images/video/fonts). `ContentTypes.compression` says which. Every migrated `Content` row with `ContentTypes.compression = 'brotli'` is Brotli-compressed against the single shared dictionary in `CompressionDictionary` (see below), converted in one pass by ADFA-5153 — but plugin-contributed Tier 3 rows (`PluginDocumentationManager`/`BrotliCompressor`, see below) are plain, dictionary-free Brotli, and there is no per-row flag distinguishing the two, because a dictionary-compressed stream and a plain one are not distinguishable at decode time by inspection. They *are* distinguishable by attempting the decode: attaching the *wrong* dictionary decodes without error to different bytes than were compressed (its backward distances resolve into real, just incorrect, bytes) — but attaching *no* dictionary to a stream that needs one reliably throws (`IOException`, "corrupted input"), since distances into the dictionary region are then out of bounds for any spec-compliant decoder. `WebServer` relies on exactly this: it tries the dictionary first and falls back to a plain decode on `IOException`, which correctly handles both dictionary-compressed and plain rows — but never rely on decode success/failure to detect a *wrong* dictionary, since that case is silent. Content over 1 MB is split across multiple rows: the first row's path is the base path, continuation rows are `path-1`, `path-2`, ... (`languageId = 1`), reassembled by `WebServer` before returning. - **`templateId`**: `0` (or unset) means `content` is legacy HTML with presentation baked in (the pre-CMS Release 0/1 format). A positive value means `content` is JSON *facts only*, rendered through the matching row in `Templates` (a Pebble template) — the ongoing move to a proper CMS that de-duplicates presentation across near-identical pages (e.g. `sin`/`cos` docs). -- Two data defects live in the shipped rows rather than in the schema, and `scripts/docdb/migrate_content_to_dictionary_brotli.py` repairs both before it recompresses anything. **Chunk numbering:** 14 of the 19 chunked items number their continuations from `-2`, not the `-1` the reassembly loop starts at (ADFA-5170), so those items serve as their first 1 MiB and nothing more; the script's `renumber` phase shifts them down. **Mislabelled types:** 74 rows holding GIF/PNG/JPEG/QuickTime payloads are typed `text/plain` (ADFA-5221), so they are Brotli-compressed for no gain and served as `Content-Type: text/plain`; the `retype` phase stores their plaintext and points them at the type their magic bytes prove they are, which -- since those types carry `compression = 'none'` -- also drops them out of the dictionary pass. Both defects originate in `docdb-studio`'s import path, so a freshly exported database will carry them again until fixed there. +- Two data defects live in the shipped rows rather than in the schema, and `scripts/docdb/migrate_content_to_dictionary_brotli.py` repairs both before it recompresses anything. **Chunk numbering:** 14 of the 19 chunked items number their continuations from `-2`, not the `-1` the reassembly loop starts at (ADFA-5171), so those items serve as their first 1 MiB and nothing more; the script's `renumber` phase shifts them down. **Mislabelled types:** 74 rows holding GIF/PNG/JPEG/QuickTime payloads are typed `text/plain` (ADFA-5221), so they are Brotli-compressed for no gain and served as `Content-Type: text/plain`; the `retype` phase stores their plaintext and points them at the type their magic bytes prove they are, which -- since those types carry `compression = 'none'` -- also drops them out of the dictionary pass. Both defects originate in `docdb-studio`'s import path, so a freshly exported database will carry them again until fixed there. - The `UNIQUE(path)` constraint rejects any duplicate `path`, regardless of `languageID` — a second language for an existing path isn't supported yet (only `EN-us` currently exists). Getting there needs an upstream schema change to composite uniqueness on `(path, languageID)` (see *Known rough edges* below). Dimensions: `Languages(id, value)` (4-letter codes, e.g. `EN-us`); `ContentTypes(id, value, compression)` (MIME type + compression scheme, ~30 rows). @@ -64,7 +64,7 @@ CREATE TABLE Tooltips ( ### Supporting tables - **`DocumentationDatabaseVersion(major, minor, patch, who, comment, changeTime)`** — the database's own semver (ADFA-5220), replacing the heuristics that used to infer the format from which tables happened to exist. Append-only: each change is another `INSERT`, so the **row inserted last** is the current version, not the highest one ever recorded — a rebuild from an older content set is a downgrade and has to read as one (`DatabaseVersionResolver.resolveMajorVersion`, which returns null for a database predating the table). `MAJOR >= 2` is what tells the app its brotli `Content` rows are dictionary-compressed; below that, `WebServer` neither reads nor attaches `CompressionDictionary`. Gating on the declared version rather than on the table's presence matters in both directions: a database can carry the dictionary table while its content is still plain brotli (every row would then pay a failed dictionary decode before its plain one, on every request), and a migrated database that lost the table fails loudly instead of quietly. -- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows, after repairing the two data defects noted above)(never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). +- **`CompressionDictionary(id, data)`** — single-row table (`id INTEGER PRIMARY KEY CHECK (id = 1)`) holding the raw Brotli dictionary every ADFA-5153-migrated `compression = 'brotli'` `Content` row is compressed against. Trained once, from a representative sample across the whole `Content` table, by `OfflineDocumentationTools`' `migrate_content_to_dictionary_brotli.py` / `populate_db.py` (`scripts/docdb/migrate_content_to_dictionary_brotli.py` in this repo does the recompression half against an existing dictionary, for a database that has the table but plain-Brotli rows, after repairing the two data defects noted above) (never retrained after that — a dictionary-compressed row is only decodable against the exact dictionary it was compressed with, so replacing it would silently orphan every already-migrated row). Shipping the dictionary inside `documentation.db` itself, rather than as a separate bundled asset, keeps it version-locked to the content compressed against it. `WebServer` loads it lazily -- not merely from starting the server or swapping databases, but on the first content fetch that needs it after `database` changes, and only when `DocumentationDatabaseVersion` declares `MAJOR >= 2` (see above) -- and caches it from then on, reloading again only on the next database change (a swap can bring in a database with a different dictionary or none, so it can't stay cached across one). Per row, it tries decoding with the dictionary attached first via brotli4j's `attachDictionary`, falling back to a plain decode on failure — needed both for a database predating this migration (no `CompressionDictionary` table at all) and for plugin-contributed rows within an otherwise-migrated database (see `PluginDocumentationManager` below). - **`Templates(id, name, content)`** — Pebble template source, keyed by id (and by `name` for well-known templates like `bookshelf`). Referenced by `Content.templateId`. - **`Bookshelf(contentID, bookCategoryID, title, description)`** / **`BookCategories(id, category, description)`** — the Dynamic Bookshelf: one row per "book" (PDF or similar), linked to its Tier 3 page via `contentID` -> `Content.id`. Two DB triggers keep `Bookshelf` in sync when a PDF row is inserted/deleted from `Content`; `title`/`description` don't come from those triggers and must be set by hand. Non-PDF books need a separate ingestion path (plugin-provided, e.g. via `PluginDocumentationManager`). - **`LastChange(documentationSet, changeTime, who)`** — audit trail for edits made through `docdb-studio`; not shown to end users. `DatabaseVersionResolver` reads the `documentationSet = 'wholedb'` row to report the DB's build/edit stamp in debug logging, falling back to the most recent row of any set if `'wholedb'` is missing. diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 7a592d0703..0926bc4864 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -11,7 +11,7 @@ fix is to store the plaintext and point the row at the honest type, whose compression is `none`. 2. renumber -- chunked items whose continuation rows start at -2 while - WebServer's reassembly loop starts at -1 (ADFA-5170), so they + WebServer's reassembly loop starts at -1 (ADFA-5171), so they currently serve as their first 1 MiB and nothing more. 3. migrate -- rewrite every `ContentTypes.compression = 'brotli'` row so it is compressed against the database's own dictionary rather than From da69775e4c8f4e6f8b44276810de96e3483cbefd Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sat, 22 Aug 2026 00:12:37 -0700 Subject: [PATCH 05/16] ADFA-5153: Fix seven defects found reviewing the migration script --limit and --path did not reach the renumber phase, so a scoped trial run -- the first thing anyone sensibly tries -- rewrote every chunked item in the database. Verified: --limit 1 --phases renumber --yes renumbered 2 of 2 items before, 1 of 1 after. Every run demanded a CompressionDictionary, and a database without the table crashed with a traceback instead of the intended message. Only retype and migrate decode, so renumber now runs without one -- which is exactly the old database whose numbering most needs repairing -- and a missing table says so and says which phase still works. The phases-without-migrate path returned 0 whatever it had printed. Failures of the work a phase exists to do (a decode that fails, a renumber that cannot proceed, a verification mismatch) are now errors and set a non-zero exit; observations that do not make the run wrong (left as text, a name disagreeing with its payload, a chunked item whose base row is not exactly 1 MiB) stay notes. Both are labelled in the output. Worker dictionary files were never deleted: 160 of them, 40 MB, had accumulated in /tmp from earlier runs. Each worker now unlinks its own at exit. .bmp was nominated as a candidate but sniff() had no BMP signature, so a real BMP was reported as carrying no binary signature -- the opposite of the truth. The mislabel sweep used startswith("text"), the same media-type boundary bug fixed in ADFA-5241, which calls textual/example a text type. The candidate SQL had it too. Both match at the boundary now, via one helper that says why. A dry run printed "renumbered from -1: N" having written nothing, and the wording had the direction backwards. It now says "would renumber to start at -1: N". Co-Authored-By: Claude Opus 5 --- ...ntent_to_dictionary_brotli.cpython-314.pyc | Bin 0 -> 51192 bytes .../migrate_content_to_dictionary_brotli.py | 157 +++++++++++++----- 2 files changed, 114 insertions(+), 43 deletions(-) create mode 100644 scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc diff --git a/scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc b/scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..cd20b49529d24d4708419767af2e6c7a8c19516b GIT binary patch literal 51192 zcmce#GcU0fsP)Bm z`{tYXJNH%<3M902l6fVuP|K~moO|xs?m721<>fjA9M`&Ty!83k1mUmgMm;S2!tf|+ z6od&u65_&nK{6aQ95Kd?Rdj7UYKoiKuQ_gJzm_A`xK(*)i`&>Ud)&@`bK*Jd*AaKH zUuWFOeqC`F`^}B#;@5OE?}$6@KH`acssw3IJYO=G3X-K%81cjl&e?nF)#{~xs|Crr z*}(7PMbh4Qv9vE72@ zbs@W6D;43om|d@vN^o7uuB*m?D_)$F=fs=@Voc3mgc;<}Dq*Gtby^|))GyAeaYL2AUEm%VG0HsHF6 zU3;ZwT(_|64U!Mnt?ar<+AnRy-6p!j_svop?%LVA7HKoCx3FuUv=!Ie*mbM49oIY9 z^+stYu6MEPP3PLA-P_L!La^K-1iz6-nI4~RtkHzAYUW{D|$jCbw9;9rLzt7?5y(EW1 zV)RlV7K$}fQSnRRm>7|RA-P$+7#a+W$3kK#FnCFfjD^I(OM$WBP)xjbDZsNF3%wj? zZ(^ZPY^TE^ikp0>R{VN2B#L573$H|+k`Imq!lPn5f?UNoeRgp?E?$a^1jT@O_-JQq zPxs;0V?7;*TThLL2QT-AM?=VM#Ssl$ABhBlV&ncpo%>t1w6$&W`o!K#AwVw}LIKn~ zRyNvjfeLAjqSUbq&8TrCmL)?4R=ucijOr@J0@v~BMFvTH7K;$%u8Y?~GN2R*kHtp- z?UC!NtETn@YOo0)a$xW!@B%u(fuXPrRNo|?2p^lJz@h#h=vOsM<-*a-DnU&;?K`z>SC=qb-Ep4Kc@s6}?7? z-M~jNqi|w7dSGQDt!rbUkr1PdVk|rw#TduHi^hOsA!AsLb8e%luL~*~!GQh`- zK{4cyAV1VlL06x6nvqy>d<HY?@ef)95*2ke#6A%|LG(Glv}UbGB^ZCH$r1C<#hsKp>_0Wl6< zgnEsGRRuv@<;Zo`yHqYwWfoBDMm zG=%nyr~|$xz>89eVz36;0+kp!9VN1Q4blkx4udO%$3hCz#FH3KS&Zczt* zQ0-{25t-zQ(5M2cj+X&98bgE#qmXn!gZcQ}gz-6u&ItqgL$O^V#(Q{Bym%cq1XV!P z2S8KrE7Vs<9T@E)il7P!fx-AV1AZ8lBa8!Rv_xY8^l5B-7}88!se6EdpI)>oT?;ZB z3k-u85$PhNlta;o46M@AZP4WL7@o0lM4cgv7ks0Us~1>yLtuzNW&r=f5UA9Z_ytju zZ+NYao<7-f;6!I9qrojL__Bq_8Aj*eWf1F7{8|X34B%t=asDn|xC&TATCb3(MCf@z zY=oEx>VvvLNQtPVfiW{NJR}A##?S^Y!$CDKC6m$U<_pLlsDAL0jQ$uOrKXOAFNa3L z#GAk=R7ip&kcAya4!%wZ7k;#@kQLbh}{sg1%6;SK(64> zkBlKd;!srUm*+5!EHD!GRt&5K}0 zLttpAabQr6Kt~va7zD!XI0K{(?PTb*o}d#&s-1jj5L1flR04(Uak-!2D-)}pW!_7cW8>k4AaL(k)a@>1Z0l?;=$ub#m2pRpnPrg68w7h z_a4~^nkV*L2fG;~b&_Ff=;bJQ0WblriHth5euB)x$PbtXiI9X%3KP2s5gLjk@d3uR znCQ#6S<4}x!)r)617o1AOdp7KdyT0a0uWTOSS)41@Er*blF>pjTnJ`dn{fL`#KnZ* z7tR~Ee*sD~o>dB`43`X&K{84v$=nJR*=zX)nHy4;tAUa6P|5_FlClCKU?(Z_#Ykl2 zmVgRaF&;+)@fa0RlNxNj6afJbqF;io`^SQEBpf^)9}k8j5YJu$qQ_bfqKPNRdM||z zM_Rcs7Hb7^1~0ZgBPyp{0|QE{LjwacG<31k`4ycte`xyG5^0E$C2hOu&P6u7{ZGV12W8v3HfdjWC-lwtl2 z$TvL&nQ9Ha9DuAGy0A+d9EzfIk?}lAcEJ$|28o`b0@p5a*$JNw#(8e^n9I;iG#QS8 z^ef8y1xSa)r{!@_Kd$9D0?{bg$3ZT85qmoV+D=4TwFzv7O~8>dQ^^Snom+tm@; z6GA_3`O!n8aK!?S>Pi&Q7!lll1ViK?lKG*#*O2q0nI=b zHTg|F)XEiyUr-SprZY4B+l+eBfuUiHS^5bM?C7DfxZR7WvY@y(CJ4U&=Om8M4P{G-Zs$K&XOH5XVv$W_3t;1`ZuP*n9M3_kQX50V3EbD}>&c z!|{||4h=(?3CSt<0MurP-Y@9%cizYZ(hm~ z7jwj!gUfd3YdJS^rZz0v*M4Xc9M$iaRwT_uAJ_$p^NS}xck-(zXL{c?Z}>15ukg*s zF~Z6p6zp%e{J7nDz?AbZA^dC*$ka8molb-m@)|ll4fS6^A`SJ;hTMk&PSfij(9cXQ zyIYO=1U!cNH0~#ail-uf21*KqNNLo!3PH>&5ZzvjL9$*53mlDZ8M?hDxe6r^7o%=X z*-oDBkmZPsu}p2%(ImMJ_w2+PRLko1z-Y4M-nTnm%Xu|ts`l61|8(D3@PS~kYfi6e*5EZ|bOMb@)=Zw=poRb2tj*-iTJoZ8)XO zLGY;fY(?$?~R|AWn1J?rvUm_DmdFcDi5N zcVpjlVBT3ZQ}peMw<>0X-@Sb6@{;eldEaw)O7D6Xe7#G)7v_C0Ecp79&iNm5kc>X_V*zQBJMb|e zc}VWWh1bZyDc9gyK8(`}D3&NtB+*i*_{!Z#${MjA7ZZX4&&B$!x6R42J+}eX!%u-K zQ5J@`)O}CVl|Ze{l2x*`LZb27SKx2TrkE2_CfH+A)@x7&BiA@XZAZpW?TN1-QOwJ# z@FAF}l~|~idICRl2ieb^LiUq9dV#&<2!g6>2hG2;rucD@iPZgCI=?{`^CW}rsix&f z#x##5713u;_aPc?Nf=wM;P0Az0)?=?!`%?55)B6~Mo=f>0UO*zd}IkHg-R*9Knd4@ z2UhlHFH$m4wBiE&=`2{z$^F&X)V43ZGTr;F_L-~S+%;!j+}e59d^ec1pI)}RA6ZN; zD|na1`Y#+Y-mEl?$WhGT6WMtLAO{Q{S`(y zS4^8J59N?*ci@Wa>@<}P#nqJY-pnS(kwYb%8rXYl#(SIYy`8H0>@^1fR4eGTfk9z&--aRBBJ@x;*+9enzn*#nr>9ap26 z7`qCi5eOa4utU(0#fAnlDvytm2g2|e%#A_c&hB3j`Rz{RXw!6fL`1ARM&}!t;LmlG zJJk2lv0=QY(cjHR;Qw0wUmrX8r!PfA^v=xQ(OEj$*|}3D;rtgQ;C3?_+}ClSC!u_C zEp#!uvzw8=?iiu(ffH&u7e}KkIT@i6E?&UPEn%`zQ`U7ohdL8(SQWToI5O4>WA0A* zMZ97K^0LTw&yrt>i{i)b71sBUbNB;T(pjQwCp+5`jv?46TCYWhhT7DJorxUnA$xYR zxA$bi&Ynf$L&{Y=KEz%Mf!Jumqp#UDJduk~Gp{j5u!h^j_H6;}a!0^x;$LFevI|U2 zg!n~#a7d-}ME!J{2Bf#K@X(MvfX9ql`~xl~g!gTElk&g$RI;Rb(boK)yL?%_->_)g zkiMUaEZXYcb5|&@r-m18YiAl}Yk#|CZrftbPTiZ(qOBtRU9!ZxX!E}3UaSA`J-0}= zv48W*)U`$1y7z0FW)J=LiMhVT+TBUFn6#g`(}Yvz7a1j|^Jf)#&t0Y@PL}LhwCzdX zPrbBg6W?=Jzvs+NlfQqKIFp??f9zzs5>tboE-C(idSo2~`-g(Ta2^^Eok>=t-UPMHcAX4(sd;$RqOGtioFj5MSnqGvO0TBDT_>=Ln+e;3?R;fN6vJ2 z4IFsBx8pR#E5gT=AjF;{Wsg$+PE@=5rt2bN0r_-S6wA`;F#7 zTxbWHK!@ScyEsn>=Y+UH65_`5|0!;&MZ@A|$#~8pnd1~ zG1()PLrDk71SOwB)l)Wh9}cFRBk1gLOfn7N#;r{Z!MI8foMaIjfvpQSjGCof`Xni* zZZ_qXA>~%4$p(gCGLFdC6Xk4ZafJ*bEqSf+)46l0T*^tUYb@npHDaSM!w{kfIEF`+ zsQ2SU>sH|NEfW`~bC8ZTQYE6qhE|$hQ~3m8pMG-S|E6SQvLhYPU^zx38|Udk{=0fa z6M|wy=R6`K2Gxwj)RjkiY)e8Zr7^EeAqXDLtg1@Y`tb~eFffRyA5sqOGz2JB4aW0? z&?|I8ZYMRf3W}!R__8ozsS>)-a-src+1g?X8)W)`L&cOb`Zmf(ai21baj9ChN9OE? z3jt=IycaAW;e|Se*4tI z)*~M=p7yzdA{e;jaVpOQw-`B&DDV?N$^k-BUsRrPB?!wnTbK^7L zDx9x9Iu~5pdgR{LBMY@h7pjlG<35_SA5C|d$}FB4mE>zE;YsKtzl;R;s4i)Q>0k)c zp&(3$AzePwu|p|8=h1oGJ`Q2Xx<*G3<_MBovmw=aK`-Z?H|X!y3+FKyU|ArtdCR~c zEl=P&1I*0*Za}ex^Rb|RG*9WdF8h8{>r&Gm{7*LRnT#*FtLNR-Gu7|7YiCZq<8DaW z58c^>Q@XLA7}&rID~e-kZi3q_K_#Bdi3$9~0Ak>CfFzCAeKbt>O?%l-x;l8z4u$>Z zJR)WPaj+*Adj|6yjLG5)ZnwEo0EP;x!WgP43YMxcPj(#^_^|r7$QModEPmLylF6Ku zIkM~=-l143mk%s96%mTPLn(=e0I3Ir0QtfU45SY1FVNf;2bm-!E4*V z0H#uLfj7E|Va$?7sSQ`?WX36FCL+%9hz(#njE>|zcU}G*rTHSAzC;O&2OtGtdUyCzN~3tJYAEW())e9rc^%~K-xi&z+rgw1)OzqM=Fcl zK@y0W`ctwfk_D4J&eBWgpCr|5K(PQOm4 zH|RtVl)pkJQeEWVqSKpn`Z}F{gH9_j<#l>SrxgBThj1cH*~Br$<^Fuva$)hr;mOkr z=HmCtt6?gC z{YC_VclMIGnD-!Mn;|J$@ck=e;n@P&qMp8PC^-s^u;kWjaJi(51z5FH}#I@Q=&TWKmI6#VB ziIazz%d6s<$6R-qYiLkMel^!jilo7eE4glwa{%}9XYQ~R0Yly`K91sBkbTO6>AQ&h zaVwv^AO{>wU%+vQ%>}_@a11s+vVe219?ehrmXVKn8tEzXVrENkXgh6T+x(}rdW6w4!@X#gj|zJu9(Y^5Lc?khHatLK}LE;!) zk?6(fIpaUfip6COG8$n1`Yn3@f8s=}3Ct$QOFcr5+vu_RGG;_#gvXevq(kwbP#hx% z*f{+_!zkK6d$wK+|8(otJepXm~zqER$_S?RX*GQd|)!H={LaM zlizRnCx{S*Ws|m_J3UjSU)noy=x0v%eRtV(?KgaH_-5;Gmn^Sup6y*~**o8|H`%-| zxo;#{J&FuG5y~JZV)(!+xJssiuaDgvo4It`{FDBq^!!4{3(ITQf2V!++P~k8w5}aU zYT3a_2Mu_YK+u3=1aBj2>Eq|}h~P;!{Mn@($ssvgt%S5(>}EhGafd;kQ@$R$Wo$#j zRmx4iv;*Oxf!HWH#L6CFi5tjQ^z4P+xDieZ6f8-cbD-M&I8iiB742jPQO#K>nO2CW&1wo zH>vhbrfhvWnx}&}=C?DrQTF@^E;;@@m(XU+fR0UTXRg{#)>EHJmk>l6wC10-UxUK^ zY^|Zv9Qtz7+%vd~)fbq^55HEf-@5&WD52?@+T&#Ka_KNt%LiO7*Kfr+ugxO4(;Voj zxZ~+B&Q_~zWuUL}(>Z$!0$;`O1$%>630M}~zC)uFprcy;$d^4MfKKB;T9^ro+d(?` zvmYdHW5iLBV?XqhAmSDf!3YB*7#R8oR5lzCP=g5b9ETZ$dpPH=qmpLzI9L^hePdN?ev&Pl`1_)Gh)Z zxT2=EI8F^5Vxd6@zGbs}#Ny!!Br<6LK=3&jHB8x*$RJqf(>(fH=xq77>GXfmiA4Sk z9!^X(+UZq@m5XyU44l+ewk?U2Agw=XFm;}R5iX5XlwKF;92S~>_=NB>l;M-gv zHsBHr7bi-o6>u2c8j z#frQ!JU5i=xVq@Rmb71kt7t>}+~%eBL-Xy2mg+n1)pvZgZaQbNpyqDdMAru{VU2Hg zXrXk=yt!!N@Xu^{Q`;5`8`4>YZ(Ec8z@qzN(th#hRa@r5ck36bPR*MuCk{*vQMED( zJ^W&_|6=mR!DL5p(H%&I_UBu<=b`q2$^jlE5x83G5P+zzU{nU-#Yg&20Ey|DEQAEhm@Ds=gINeHymkX(9>iB&D^S z)Y){=C+zrVE1eEZe~j@U){hI6xE9|*?HO8s9P zgGJVkf%&rhHTj3~e@{<;_Gqj=U1D0eiMsu6f&(4FGsJ@Z?rgs6=QXD$zj+6x-N({e zv-yZVH3wUNS~$yInx?#~aPzy`&FVEKZ>|f1PEEG^ZT=j;!|(Lx`t$rS7i|M1=;%9J zk8{@)hne3BIDz6^GsBJC6SYP2qb|RtM-%zjC+y_+sAWY9`~^ekvr3_>3JQ5aPeD4B zdX*IhHh(_qto1wU42V(Bp+rsGP;*ExM1mTMs9mAIFl$Mr=`YkP$%Mr`%&L(M<<#(U z`peW5PcD&`Yr*#lZfjHDCQN->6MxSFz=#f??l|6Yuva|j3nK*ippRxt=~ta0J$i`l zl=)8zXW2tFaq)!jDyULzw;m@qV54qeZv5*5d)1nHgADEzKG;K|yI zXE5PWtMT&|estB6{|oNr|BBN~x`H?V{@&l&%L?}9rfi{?LxY&mNZE%%h{b{)oN^sJ z+1-1z`^vs$%0|IZ^oTkzWrgF$ zFk(#{>?tdZ_vfF{2dv*KD4hi{{*WdaET}^>BoYOSxnSZ*_m%t?NV%V&M4*}wmYu~* z&b9N-wGWJzT+aiCkY7IeDKN{5oCi*8j_Z-h?y`R95p3?yUi$n?5Awme9~KE>?SiZJ zjrg=_D*rYAr2mchL`Tw9yS%n)qGNf>wuuve-G5=;es1b<33v%VGna*Rpd(^#}Pm6?u~#U+Q}3 z7S0(0WJ>n!nzwr=a}aH_b!O|br}*`pn>o|o1yAjyW!dRjbXG3A@|Rpy^RB8Tc(b_b zXRUKq2ty6~-gQ0q(1yPMcP-h==k4W7_NsaOuicrnSG{ZB_0S~b zSCK8KV5;u*=9|r5@y*xoUvwTwnh$*R%RNGF9dn17&F`o;|Et)%IkNpnLI~=AX8h(B zodDf}FVip~0e}sY8iohyptVs#01#my067}BkS}oylPvH%*Gn$OEw?YTr>n&Z8rNVq zhDXO32Kfyne6KtZ_F>2t%m(4X2=ReW(IO|Z8bsmAbfpcGw6L-$leCzcr4g?lQL`lU zXf_`8DJJL)I-7^ae4BMCH704v(6I?XD-Ox%w?GgwwVC{8)$#vH;UikkR==5ArTBIs z=PUJoqkc*sEmK4y_;bODmw9fPqG)57U*joQnK@->P8s$kS`&<+@187qk6I;~D2G6c%J{3rDAPkAavqZr04)6=6k!C5^ROS!-U2Sa4phHi{- zVp4ZjTWS%Z0;137SpF8@GCZOw79u3>*LL67J?Wc^FKs(|Z`;wuievNJj=uHE%+=W| zbJbt##Lc~mWA8YQO&nYv0cIAmHbjYA4A1qpYif7R19gE)=eL#)yt3hjoq z59ze*sd-?+@B5Ct2NwLou${Z+fsL;0g1hiR4qZ8f67hkPu3Un(_+c)t`1sb17z{yW zMynCffs|k#0`@}a&N|Ld7<$c73+$IjyYj$@GG@BE^qtD)KLrs~5$zCUYSKh~s%P zA}d5KetWw?2d!KvU+5k6h~{=xa~cZ8%jvfvw-{(Fpez3$erHxSB1h8lL#g(m4pztP zrCR~FEXoC_CX^f4Z%uellE{jq?{Dy&9IaLy?OK=~`rWmnC9bXkC^OmXVMLC0J=SzV z)Mbjx|1YrZ`!fmyV^rK6br3dydfO9+``8j^ij5U>{bN|ANLkg176Q}75R!aI%Ym>} zWMGY@Ti8;u97@pZ>rU)F({pHlZ-=TYK(Xo7IKduqpT-ur5(CX)pQ!N!-0YP9Ga%jF zO%gl{#!rZcpc)~=4An&cN#ZM-ZlUTwN4rmVNWI`Oy(hJbG;*CvuTd!+YDJx}V3np; zc~@oa-+%l}$7!(ldyF>!J-uT2zDHO84^D{I3SjCsC^NoD@-=6Uc7%yzQMR$+lwIww zluhX>HmMn-!MwsAs7T!`_?%B_{t}58+5UCVZWseSLc`X3^?N3ROL;Z-@@nqvn>f5| z&Ra5<%$rM=?MIe#+*5`xmrWi1M*kcAN&AKmEC_A-&><9*-`GDLU#i+TU$t?f<9=@b zSH07nGi_f#HXB+j*u0Pn|C#0OyO*{fySM$=JMOOe?Z@WVZ=TsYw`tn?>v_1FIC9V3 zwOm%7G?z~t`OqcUJyVC4it6Tz>Sps7i#9Bmtb09jGxBap!(`{Ot7OTwZr-&nS+)O8 z=baskt`o50OCGh36M4gbSp%FTONCRElus1*t;UT?kG`fkB` zlx#2HB6_l7&mHUS>x=fIh!C6F^kvtpHdMT@7WHryC7qQ?bLB@L95x7^;|3Pj@#9^2 zow@c5@mv#i8_|DEVSdaSD#{G4oy7&_JYFQxBOqc(fO8z-cidfF(A2_9P^0kWws9o|uA_+qo zG6cbY;$JM6rwNl|_wD}M-FK^#_T$SrYoPf~x8sC?Xt93u$UEpWhOt^v?-kn)KAlMzehty)pIQ*DdUt}+lm2l67<}(M3zlvQzbhxolC}J2DomR5+~Hf62_3HmuIpAQrPbPei9L$SxQTOkJ{>VI8laU;6w!e28j?Z-mPZ{4 zHHiDW4~bMX2@hCHieaB(8-rqk$wr2UAELSP-{8a@p|y}9mdOBPjWPLSyk~&sdQ_TL z+YjHZy<7UoWX!exoi&dckSv150mk9VpEh#}%)w58Re`I3-S9}n!za;PzHZe@T1(_2 zQBI04EuR57ge{eU@z0mYc)E7=L&BL#R(3iFJGpKnkGtM-QWtlvRvC!}ja3BMr;ouF zEj|pohJjf^xUS*gmJqhcK(*i0ZiK$!Qmi2$V>+jGs==N5GiKnMdp@HM1|877TyESNBK)l8 zwuFSBca}%ZI0J{Xa$6`oR5j(G6`8R9$xh5$;TuGAgHWX1J9G|CS@X(Xf{xD`vaOz? zD#UuOt#VTJUDS-cb@f#1>o+r%7Bf?*t1){TZcamFbk@=_1T1H)<0dwfh^}F#K!iPA z31JQqC9*k0ThxNt!^eEzc}^KNmslDMn3V@=VD9_89?qnuMsqMcvnb(`#Sb%y3W03u zvPR4uen*2+*f8s_t|L%aVHC(PKG=2unUwcu@g7RDWNm|~unKOcOkp*uxdbs^$c!w6 zq$>`n@jyuCV09KP&0hD%7&TBL)Ox0zTiSJNa&K7nWb;yr4yDb z(wa4D0a43Ibf3NyrnPgqs}iI&lFYO?L#$AVWxZp{TJcP~rFA@8uok|$LQjC4UDVil zA9J2b)MRFY*m~v>!=4V0N3gtcVLQpqxdn%?JmoMPlHfOl8j;U01T=jaiN-84q?&1lN}tGWQB8@^MO(Z%JZ$GH zsH02_hr7=(iSkQydznsuLZ^>#f{#sFtR$KUy8r<>1?`qrBvsYkqx8xI0JGqylobp4 zVb&{Up*1EiF_sbE)+e#;C^A^$g<7(#ZQ0s z)6@1vo4B0cK6mK%PrQ9%Zr`1jq`N0+?}0+s(6m&){a*d{MaPc$`t3JIriP{iGx@J> z#m&UQdB=|BHI++iHY~2$aKklenA)^lv__|h{`UD~?dD|u7U+(iaz0g+Tz~AY|L)NR zkAKNCFz*>y@B}6;KXZC+Y?(SUz3Dg3&YXU?xN*6pa_Vey?dfD;?>$%V!yKWkiFE2* z)TgF?sk(i!x*biQ?3~`ODZ0-xEljNz-O@#7zX|lxLK8u4=JX;)*^Jh3 zAw_OQ!^dE)h46-#_eC#c)%g}grIi(kZjr0XL2i0Z>m#Ho8=SK=VgVu(h6se zL|2bY`Y&6M`x-5;eN6^oVl;|9gI-1cn^j`IL+@0zC)aJB>%6_?cEg?gWWmAzw*IFZ z{&>TZ>&&AZp@PR7W(+Jd>H|dlWD3kH1XjxsChFtdG(_n9R9W=_V4y8IWHD5bCR;Iy z#Ygco6=Qd41q&InL#yAf;}$;mb_|5w7zl3#$bx5@z=L~Z^BScV_=yT#dleyse{E}nT^*}>8V;mP zhr9&x3GDeumC$WS_V-w z?RAi`+=b=R_-J5Eu0~o0y?zFSKlf~yG~WjbT=~-0sYAcIXZp}g$D1dXy)AQv->H}i zOq-|9PM?~NPY)#v>+ZP@-#&Fae*4Is1Gignvuw|q?EP}is|L6W)Imb5yf z3e-NqC(we7I+CdA%25kuv#f9IyWM{Q5 zVzoB*h^tdcIV=Tm7|N;x-(-bf4{&JZAvxhzsxW$~Uf;Vtz?GGU*0cF=t@M!c7kq)$ zDqm{Yfy6;hM?Df-T5b5a@-COv4m}E}dwNujP?x!sr}m!uau^~LQhzd7yZ!mV5fA!4 z3nYOfTGS9mY)-ab?E9(TgLzklk|9-A9PxV|k0X8$WF?{RyFHD%<|&sMGue*5AN6c_ zN*ZT}q3cJA=!Mb34}T5}3k7F9ieUlVixlT7AzDDV;46oCKkU(kJlt5N7qEl z(rs1@7+_caGO2*!D)=_2c)QxPJ{NFGMSk@mex*_@+Y7IUl?ROl{~BqH8dJu~ERR-SWx0G^;cv(DebxdxKRe|K*?)V3J%IP#@m{%dypBZU^3aIev$rFD_kY?r?JP{ zvnQQGy;4(3WpG3;R|S8-v5YGfaMiBV^<@QUQYCm`k-t(}n+sJ!#VOPfQ_F;Fy436X z;dadrze7!l9)W(=L5$qApAAL` z`cfIw=(AdX2}-Z?my!~seoqtuJbYa^>5>gs!$I6O1= z7^5!5_ss;@6=>H;1tX9*@z*fs(1Zn>sbfK5ut7X_PeY2=p`{Vo{D~6$tb|1)W=*vV zHS$?9Y{o?{xy?Kh3%^Wz#pZ?`6sHaC2eu)Krb65_g3$QxVr-`@+EUEHH&0TmJW!B= zkzG6k_8rz9fC|NT=-@G8O6V1B)W&9Su+B4AzkH5F!Hb((+s`Q*k8rmOHour?)CZt5 z5U>DoEEvZ3sze|Y-_?m}V{$I~PX0D9DBd{5Rb7^qI{mN3_cmmh@M2Q zCaioJ#Y7JFF2OWVQ1m5=*!Dj(#iWLmV^%8hToykNHklwcuAXx6iaOVh2DT@hmz2e^ zP@;@C@L+m3iI^xhVe=tTu0;J4g<_+A?t;GYCaSZ2#J30I-Wa0%_h=HNefnW89ZVI& z4LgW&15pi+$opsl3JF9Oop|Jhk|O^hUMY%J%6R$eO1JXVI@aab9e@yWwX6GcC?iH9 z=7ey^cxTgpv3&n36QJck!n2R9ya6d=TU)}ocXJ|_-}CTzF_A|Ic7W}GDYont6K=ZK zg4M*vl%a{)FPW1kI*1(%Ayj7+G2z5R#UVxf#s9*AHONgbMTZf?kF`Ak8XuuY15RF1 zE}-iWY6e{#{_OvguCS%TMKauc2RHKH(i7XzI4v6huTX@C%Z zGc)7Y=_*O5H|S$K(2GqPAw7#6g<2oMrk*nBjD8)H1-c~@Whw{eMQrtU4G!?Xq=#Rl zhkT?Bu(gY5N;_pmMW~ufl%$%PvzA#ku`>hK!Iz2C%GGqD1St#k7B+$ygd=M_mNqod z0_^lEm~!zIaj=O}JTQX2b!c52e#2avVEVpfjV^VcZ(p)of{4WQB3wfdWfnc)l#n(P{<5O?$ zo-O)L;`WAx)~*Hj@w>r$ZV4|Z4kz{?Z?9dpmmbtU(~$@%%E-oSisAbI*?vTShC z6@1_Bel71t-n1FUtkUL1dkaDtzft~1`Ap~R{)IB%T>d*{ZIjl?t2gqN?M2^kz2SPt zUOVYpwimwsshgjA$6oU#=ff7EaN`FX1$Xh=rHjQqZxLf9C4#&bxKAm8fg}hS}hyOf(4Kr*n7gsG6H_R6|EPJHnnHQ41{g}5e$(!uNOz+zDOO?&@ z=$D~n#g@sAsk$5R)y_{AwagAJ`S#BH_9k2Q-M(_C_>L`k_=RM_i|@Mn->+!UCEPL> zn(IsMKA9}&dDnI7VWCh^_Mk+lZ%CT!C%UK3{draWyuD)5Jmp`ms-5wF{nPXI%1PT) z#n0SDtm{jr(NE%iaRbcrh!1IgtNC{Mo%kPp=Et9z-{DVQihN))R_BqW{m9fXs+(Ie zed!%n{UoAJ%clF@an-(h0@1O#Yo;Bu4ez?z9##lq!-tiy(=$)8vUz*iqP^nhWmPlQ zZ@b=deKT)b92t* zi8t$JTnptb$ui&U*bmR#*|o6eWO8>8VrWho@_`W>ASy%#l)r=!#tIE3&{XcQ}7cE|^o-In2XJobqF{@GyCyR$20cDDTPA zYDBfFFyhg+X#<;N(nY`8R5(GgFzcRMba$?^EGSuYz?A3GTDqR~W=^&kS;Vns=~J}Y z>36L#!l9e}FfL^%x6ZCZ9LoN*Ux2E>W-$Y*Fd;(>l+kXgKWdQ0G}y06y=1ai@B+{L zj!)DIibc+nyOaxMH!ocVyLY!~y$iLsM`@Q+gCficP&;!ue4%!FbZzFUZ9Y`n0>G}& z@61$1`@ZjY=u}Dg4JbW=Jw+1KI{5j<+)!W-zb~f?{j=?*#zf>P4Yf~2D)e)*F`xXNYKPI3$&5M zniZSVYd#kr8{uxI(N^>$O&P!>WO&qx3vFRz;B6FBMLH_1#p&zK$I6n&3Z)~z5L)sW z>dQYMGlViQ*H7Gj3b$--M?b40e;Q9y)v7$q+o03c)D@A>5>KI_z}7l0-~M*V?EdNY zZxzm*`ew;Dc7JX6@0KX*9amKwpqxQ~SGTim*_0?*4IZ*8zrr= z9jAI(`LN>gtV>iJZK`z{i`dgbr`S33`q97T{5?7A$GLb=k9%-3K z@fWz55FR49pzvES%sFl!UaUK`Q263}-J$8;r6VuAbL55B4^17y?R?>j3$A{0TESX# z<@2`k>F}bh;k}xM1y9#obP?NDBAh<*4fQVP(O<6-`+Z3v~wYM*F@*f zY}jxPB47>Q;${Z15#Kv_+w!}|=5{Cd^)2TYzg~8;Y@U!7Yuhd4Z%=x5++GJ6(tXtM zkAflR7_Eeolb7_=W9gGE-%QMmB-?tEd1n^wew{++%AM%=b598-JKdGjFU*cFxVKE2 zvFP3Vt97tAa+>#+W|Fx$WsvX}y?9I#XI=zaz zvs_sE`rey+lLa-&n&<8m{ZZAAtCFXr;w9t{hsi1w~RgH z@-C{V9b|dJz2#8>&J&nKjhb|nHdEd;D^sOzNO^~{$-90tra&QYFj>bD*Ilev=ExHO_cKSCVC9mPQL?Zp4M376)FeIy3TVQ(ZjRdfZZIc@TDtGe z+Kygohpm@S z-ZNI^>M$c?azkXY0cAMZbe| zOO|7=gR^f&^N8G~XhC_17g*oqExb?JDDDvx6nue`Hu~;w=wv`zW+_iQk#bg@i)Hk} zL)R4(yF5VYh$PY~GH~oFuben?f8);EJMUau+<0;!uV;Sa$yxJjzjEVOrsMbA>v2am z$DaG1lBrKGdFsH6+$Ch6oZh_P78z%Hb@#HX=(UqKPLc=DOn0)febKdf9&)HOuX*>}y^XA{XCXOqvLUwZy>68|q=PM#e}R*o*Z#=s_*oE7uVis`G1 z&N_XqPRupVl_z&e=`ctsn5+yfx`x;{Yq3hpJ?C01WZUvq%l9__&~iJF;VqkdvHzX@ z{R`XAEwrCssyVOAIa#~scJN1|-ygj_aQ9@gvVX~S?qQXXU;jWv805sskAAscaG&CV zTQ`e4&6#T?pc^vX(arTCE z+Cg&$Aftvy{}pOypdP+Q+N*2~48|Ui6@KVa5?^0&NP;?x8>J}ooE5KeD;e09wFChi zAlunfvDL`2DFABu#B_Fqh~_*)IxO#-l>xeV{#b?!qk;~}^{gpDEOW7zcHUF#=$2xV zN6ME9wkc42Y^w{UxKtz+>r?h-hfhh`k1D-#Ef@*)EIYEXk!sD{vzncr*MbLia_9RQU{Ftkr!XNb8>Aiz-Q!s-K}pYIHf@mDWE)&b7d^x)pc^ zA#CNbS$PntKV6^Aq?3kw5Ise^jv<|9L`vtTiW3{4nQ=nx|ARHDnvjST$K0#Tc87+0&#;Pp5@bdcB%=HV-Pv zt5UTp;XHh4`vFsa^yqCZ%Cl@u7( z9}|M8BipC0xs*!C3Td@ZJ3-f0^efaQ?0MC4qJnx(@|`HH7Nlp$Nv)HbTX2%~#TK=0 zaQkmn1yME4F#O;$`fKASgYRSDvq`N>&o;I6^sm}-C{WfAjOtK2teIhBkF8z%C#>Cy zd=Zep8OHSbc7wE~R2Z?3V1Y>;u5Qha4Nxj{Dw;0+Horx!X%G_?fFsRZu%myMdX*0F zS78F9?)Jy#xkK8SZM7NvjW_UNTY2Ek9=v&O<#Sr7UT@CU zL1C}oph1Bvjr-8T=N?xlwfCz?RcF4;{Rh?3)OS%w{~`5Rk9MU#5ri~%#G3D*IQ-~2 zqF$wc*$B@9fqp3P{6JWqLE5iM2D=Oc*rRgXG@$X{s7pGK9s1PS!CC6}UTloi5A{pQ z(JUQQ#~CPUt}@BC6DEUPe{K&ZDOMgmr&c~mFE-k9diA^hUiF!Z0csB@oMAg?bS`K# zTCPxk%~v~`*Wagp-FN(Sec!sxWVXgZ-fZ%Bqn^IkGNxqgShm%Z@gzsrm&1gZn-==Q^mW*{jODnRL z-SL>R!L5QXGg_6#cJN5hE^`m*jVlhdH7DG7A6W4cx9WRl3>kdVdmY6Ku1?o8y!u%> z@BXm*w(o|DktorwVhazn#tLJG^TIe{R-`uir}h+!U|EUTooYTR=F#H2IxL5la^)|2 zUQ)|V|Egsw7)n#ggrShA2t%WV(cE-@NQZUwQ}aiboS0-gqI*-Q)&W%OsNTS&%juYo zCXcGMk-ESIP-_)ZYCh`k(`lD<{JbEYz+X51PM#N9&Cw#}UtAn5VZWu(HK1>0_$%*^ zsO4g(+%#X%+69b@swvXHYA?XtXzYahB7B1j{1yI+?5Xv_jX{W3`YU;#u*i@OMtNL4 zv2^b0RexN)^5^znQ?L6@sJ)ERvTzAJeNIiA{>4*_kd?<~D}Un8g=zJ$(%Wm%+iAL; zSosZ6c>h|pH<-cIhBmD8ul29fyDx&CuIu^q>aW#)jaK=K{8c@_vO2{sni?gBZpX4atMgkQ?Roh1GefqlfDv{_>tLu1@2x zQX$4l%F=4-Bs26DF}hXuv^={0BOs6!deU*qM;Xbmq^0?R@+<`p^&4sp`%WO2SJk`p zujG?@RPB}dzl+goc)N=k&@J_^@mFU&Sp!q2gX2-UT*b@%iL@AA?q60*f0l8SZLF%} zB3kpT5{loc_ePJd_pjIb60HFLsPV6FrU??|AX*%)_19{tKq0~Lvd|5GjXr08?W$br zbae2PzlPCIDO*nL&1rwF)C&rG#$R`Sw>~e)kEeZUI*L(IqAyv+I#5jqwE7n5tSU`a zfp&F)`!PLc1#i*S?Rhl^^iW!JeBo@7bX-5$sg{qC#9#CmNhh9^7Vo?D?=|RxUuEb& z{VP*4*qqb5&FJcQ9;r|2^Icg-9fmS1g6`XavxqN{E}UHhxapz{bSVO9n4c|HVe~lO zBwf^{F3FzSU#zto!fQsm2h(3azJI0Q`QXa8`en=f#Tj@L0^STghA$8Q^Ty8#%%l=7 z>HFqvxzo5wQF=t*j)9lm;^K9QFGhhg&v|0&Z(4D>%*Zx|bJBP>L5D9c!(TX~kNxtnEN zi9%D_l7(4=N+FCW__dIG=wFBN_{Y0UYX!J8jUzzAiNzKMv6~R>qLp&fmP*)W3h@WD z(n>64SJ!?^v|@{AEOI(V+cHPSe8Gz}C&;3au)bF#ZR3ocD`=lmg|UraIh)Ifw(*#O0;SlSh9slcd=b3X-g`23z4T8Ii0a}=z{WfxM+t-h>@4ahKrc6yyF) zN>UQ{E^+-D8T`$S|cOYTL5u=Fa+o!7|G z(t=~ufEnl^;_+beO=htz6tQ*0-Da|sT0vAD>i zp|mW-Mm2T=RX|4is3k9I!#?HTAiPREcu9_ohR6A~SmDdqvT-P;8%iilX zrEG(jBH=+mg=RTF?XxuT$7RtLNJk+A>vza(F5(->}mjQDaU<-(1E@+Bjg_!PfPi0Dj=1$ktCnI;*gKlo)D*>4IF zP{TBwxvvxV(!{DSY}|tu3d+in+)uM~6j09Mo%uFW#-N>X?@Y(xw?8u*N)~zHQTDr^ zQT#IV3k|)Usp5kQ#6X)4X>d$@K>-x&K9z)Gi(ah6O+#2)7%}~c$WV;AOakW+_=$c8 z=$W%E#;?*Sb$`TNaK_|Kf`^9UCrd*prcqLiyF|OPWEfRIT{J9P(YocgH+3hDeDb=J z+vw^_u3Cx)6IF&sBA`0)nCW|LscmFyX`1lTeIRv7w<0(`)zFnSWr#DdEw^l*kV} z(MF&0q=G0HZQ#j?8If}MF7Z*5@F=54`JB0!Q`oCVVMOWY6x6D+mISRIuds@QXO#9a z)g38YpL7)4I2}%`Rpb^CtJKHHD<1+76PJiolkngSV=*AO*M}1!nU-@8)B1KXM!*s+ zjq1)`Ae0RhKHVUml(g->#N$W1I{5R}x(2TjZj63Ol;Jy-fP);-*POdGz^A^QeCkyy zLX}R~RX&wiPa+S%Gm1sFXtQkMnxhb$!ict2whizm3V2JE6mSv~GhOZpq1$PIL%m6p zZ=mIs{A>e9A?DD<88HWH5;0dd*ZLgtP(ADk6<2xK%TcuW5z0(c>E=^^yyebjY>@qK z%icfTqEP8I0^@@Su1s#;70;JoX!P z43<%E>IofB_9i@xAfqzsk?>>&UZXQvKsI}-24Ev`!lU#Sk1y6g;UWuv!WyLJjjxPY zfzm`RHrAo=I2K?_zr<@Z!68u(d7k|G2|tzn=_!(cw(Vi@0DO6X#zX^T0^+@JbR2_Z zie2C_6c8<-#Vs>Y(YteJ_b-UwcFQhwEMG^!Lr865i?=0m&=s^O0&Oenkf0v;_a8iS z<^&BYEV;lJWB}NPWQ=#n)x#kO)uOT&rdSsf8+iI%4T)`zZiTHX&VjUCC#ez~_ zRExi$6XP}8DWM%BC`Ri^r1B3O?cOgvKhSZux1;+sHo-g1g2U0vQnr)j$V zEjqnPr#I;IQ#$>UPG7?boe&EYC8Fm_gWo-J>&W+BSZIE3xoXGl^*?I-e&bJeF6``1?l_q|IhtHMhIPU%YbXLb zYIt8O*Ol}gxZCx>Xjm6DC<%;~D+YviTb#UV$u%32n>&*|XO`E-wJfgcvT%N9{r6(G zH~mol(JMcA?G?-JisjtX4{QtuW{VXIzq$&R zoR#y=$_3}Ti9@t|@{t=ymR#PX%Zud^uo=u^nRlY&{kE;Y-}H9VQrp4#wu5&LF18(6 zD0yMN?Z|BLPrLrO>!&Ba`st~0S`XkiKaGd;B`-`I`ohWOn*Xn?^N(rcJmdHs9N0b^ zW9+ky!NwQ}m|unv0tpZj2Z)JV643%_Lx{((@Vf((9}}U2wUz4rkYdtxm?cw=wNfrx zq`1?hFfG!ER4G)N)Ysfy&LmW@HPW>IAc@6NY0{qeaM`9w*7xq-_q}`1=kt5_Ja^CM z$-iCp+cI9;vRD=}eq9tf`1-X=flG^J_~q#q$OtXR9yvMYp5=}uT+J8OB4^#w>nqH+ zmTxS=n_c|L^QhT}v_BCBhNXdt$iRfC^slAro~5PF+ppVM=C*Co#;29ddiU~0AoG5K zs(>%0H_m!D6!iO(5!w^Jcz@>iGd%72aPq^+_x`6uQg~Jsp7qqskQX_RN~yq+&!3$;8&gmaC`1#CF>1+`EX5H^F)HciEj^NDBw`uiEn@*~RBc&RJGiVR zQYx$w3TrT|Kz~l?y&(0DMS90X%{yxbCTh%O-J&r+K<`jslE`++OIl`TSm982Ep&xW zBf3POO4lhB(pc|L-E4=ztf?=K3bq5tgWt_Y^aX($*sTtA-0EEDgrjG_W9My$k+lZV zwE|Taua$qh&?&As+;@NUtI_cJ<(iegZ%4ivS?NM_r$C+CZFg_q!rSOz8?v?|x;R)1~i);AZl7B`_h1^G=|&yBlB!Qs z^`XI0{ObC}`I)&H09~wE?CXwV&WjEW2#&!QiHQ#Fnqx4;#N)IiLqWt)z#9rBLq)_; zAwU{&L(>lAwKN8)h+}XYw|_w-W~2tZfKxcsNG^L($60i)o3DN+qNxnDaG3z|5H&eC zcTifeQ`BS!ngHPdf7>+k%~FnDu6%Ul;SoNgRn)Zxli^4C)U&BJhL~IS1-qbjt`{F( zDq8MaX+k}NeDP33;|#b1)1l7)-)MYQ1~?z57fuW6;*DHqz`F~Y7BqHlSaP`|E;sLL z_@@kBwnj8oL6fzvNxkO%2-@u}>w{H-#<8O$EcUtmtb3=Hp!>;HRo`|!?D8+Cfc4-y z$(y|p?-xi~%PZvd#}=DHjZ*1RvGnMCWw3IwDWE|5W6yq!+Pm~c4cE=O!paARr89rB zE@{z;A*3A^lq2!LzJqrS5q!tYm$dSQt$gkqDCbS2Z4;F3(BMTg5fr^WO1_xmAd%4m80Z$CmR#m1Nd#D`76!2xAY#6{_dd>yU;%?=Bs#;hl}k zeUC;Tj-s}=q&8oq&4;@CrS40S?n^)fNz-o{%+He&($cqq3zD;?B2pNj&6>3@wM1r) zl;K*t{thh3$11>!ytPq3c7Rjj$fkR2=YZ>?;XYh6;XFA9e&9H>zrG8ba+i zGT1)RQpEK=POn%d11I4uj_NZ)l`I)@vW_nsxTG(<+yq-6PG8Ccx@;x@M4Cu*)SS)g zKc8kBk*Scg{-Ga6ei z&6+s1=%|D%Uv@RBej91e3d+vCwLQP_ajnQ&hUjvE+P@LUnPjtBD7ze$H!gQApN^mD z@^E^yovq_-<$JyDZU;(hd0QE>mLnQpJ2n)C-M+sxveV%avB1qcj-%{`<%;Fr6+oyA zh%E!+i9s}SMLcl@{p>1gnnu$0(JcF3ccJ%^@GHv*Z4!tsQM&2IgYdwg3=!~IoMpul#fjLffMk90YUn*WG;x9 z3y{glC2&tMJYS__)G zO?_5qQqt!loU{z~AgBuEOBqFxj3Q*gl7WR5-1A*Xa1XigAalB6l!gkTF*yX&x_jgqSCjCQ>kSFn@3RMo{+W~lBsRtJazGn zZRkw5N&@>4Rm!U zhso=3Bm1b-JYGeus2HpVG~>0@H&im{YE8~WY7%cfI6-IxQ!-ZT@lq&~Xm=h=qK8eO zYpI5bB;{H`GS<@LB~Y|(ajPe2`nx=L4J>WgapO(Ood}b^YLc*;4uk$p=f^^gx+%&vFmwTQ93Bo(E?})qC4u(i z$(8<|aVQ(={V8bA7@o(tJ<#z=td)3vyzc^T3KqdsvOKAP!iyL2CVvVFRo#UWImrCyMFvL*oMf3xhHxsCjmSLj_Hf8IKE9b40lb~1b@H(6WkWN zJN>dXYQuBapbQ>Hf~#77@(5-Qejwn-zDzk*-w)ncZ18~d91s)!zu@xs@pi6b$-#@V zjfvk|@w>5jv7ExvhQ)_v5X&%@c`UcE+{GfBhO&Js+lR8<7sC}lVEGponIa&Ym=4?} zrku&K{JT~vHfsF%wWcQ&Fc~k(j~{j;GU1&wuv+WDU3h7x zv170k=Skp&C#ZK4(#fM+1jvg-Slp5qtKN&$mG4Fg*?BpQ4-Px#u!L-7SqZ50GW$+nn|2gA*Kgjd`u^Ez+*iL t1!Q)tjfAN4m bool: + """Whether a ContentTypes.value is a text type, matched at the boundary. + + `startswith("text")` also matches `textual/example`; the database's bare `text` + oddity is why the exact match is needed alongside `text/`. Same rule as the + app's ContentTypeHeaders (ADFA-5241). + """ + return value == "text" or value.startswith("text/") CONTINUATION = re.compile(r"^(.*)-(\d+)$") ALL_PHASES = ("retype", "renumber", "migrate") @@ -104,6 +115,17 @@ def _init_worker(dictionary: bytes) -> None: with os.fdopen(handle, "wb") as out: out.write(dictionary) _DICTIONARY_PATH = path + # One file per worker, so without this a run leaves `--workers` copies of the + # dictionary in the temp directory forever. Survives a normal pool shutdown; a + # kill -9 of a worker still leaks its file. + atexit.register(_remove_quietly, path) + + +def _remove_quietly(path: str) -> None: + try: + os.remove(path) + except OSError: + pass def _brotli(args: list[str], payload: bytes) -> tuple[bool, bytes, str]: @@ -139,6 +161,8 @@ def sniff(payload: bytes) -> str: return "image/jpeg" if payload[:4] == b"RIFF" and payload[8:12] == b"WEBP": return "image/webp" + if payload[:2] == b"BM": + return "image/bmp" if payload[:4] == b"\x00\x00\x01\x00": return "image/x-icon" if payload[:4] == b"%PDF": @@ -262,7 +286,7 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only # Decoding every row here anyway makes a mislabel sweep free: report a # text-typed row whose payload is recognisably binary, whatever its name. - mislabelled = sniff(plaintext) if item.content_type.startswith("text") else "" + mislabelled = sniff(plaintext) if is_text_type(item.content_type) else "" # A stream that decodes *both* ways is one the compressor never referenced the # dictionary for -- small, already-compressed payloads like a 1 KB GIF. It is @@ -409,6 +433,14 @@ def retype_rows(connection: sqlite3.Connection, item: Item, content_type_id: int ) +def table_exists(connection: sqlite3.Connection, name: str) -> bool: + """Whether `name` is a table in this database -- checked the way WebServer does.""" + found = connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?", (name,) + ).fetchone() + return found is not None + + def content_types(connection: sqlite3.Connection) -> dict[str, tuple[int, str]]: return { value: (type_id, compression) @@ -452,6 +484,15 @@ def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> st return "" +def report(errors: list[str], notes: list[str], limit: int = 30) -> None: + """Print what went wrong and what merely deserves a look, to stderr, labelled.""" + for label, entries in (("error", errors), ("note", notes)): + for entry in entries[:limit]: + print(f" {label}: {entry}", file=sys.stderr) + if len(entries) > limit: + print(f" ... and {len(entries) - limit} more {label}s", file=sys.stderr) + + def human(n: float) -> str: for unit in ("B", "KiB", "MiB", "GiB"): if abs(n) < 1024 or unit == "GiB": @@ -460,15 +501,20 @@ def human(n: float) -> str: return f"{n:,.1f} GiB" -def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[str]]: - """Retype rows that claim to be text but hold a recognisable binary payload.""" +def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[str], list[str]]: + """Retype rows that claim to be text but hold a recognisable binary payload. + + Returns (retyped paths, errors, notes). Errors are failures of the work this + phase exists to do; notes are observations that do not make the run wrong. + """ print(f"[1/3] retype {len(items):,} text-typed rows with a binary extension") if not items: - return set(), [] + return set(), [], [] types = content_types(connection) counts: dict[str, int] = {} - problems: list[str] = [] + errors: list[str] = [] + notes: list[str] = [] retyped: list[tuple[Item, Inspection, str]] = [] before_total = after_total = 0 @@ -477,11 +523,11 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s item = pending[future] found = future.result() if found.status == "error": - problems.append(f"{item.base_path}: {found.detail}") + errors.append(f"{item.base_path}: {found.detail}") continue if found.status == "keep": counts["left as text"] = counts.get("left as text", 0) + 1 - problems.append(f"{item.base_path}: {found.detail}") + notes.append(f"{item.base_path}: {found.detail}") continue target = found.sniffed @@ -490,7 +536,7 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s extension = item.base_path.rsplit(".", 1)[-1].lower() if extension not in target and not (extension in ("jpg", "jpeg") and target == "image/jpeg") \ and not (extension == "mov" and target.startswith("video/")): - problems.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") + notes.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") retyped.append((item, found, target)) counts[target] = counts.get(target, 0) + 1 @@ -533,12 +579,17 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s f"({'+' if after_total >= before_total else ''}{human(after_total - before_total)})") if write: print(f" rows inserted {inserted_total} deleted {deleted_total}") - return {item.base_path for item, _, _ in retyped}, problems + return {item.base_path for item, _, _ in retyped}, errors, notes -def phase_renumber(connection, args, write, retyped_paths: set[str]) -> tuple[int, list[str]]: - """Shift -2-based continuation numbering down to the -1 the app expects.""" - items = [item for item in load_items(connection, "1 = 1") if item.continuations] +def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> tuple[int, list[str], list[str]]: + """Shift -2-based continuation numbering down to the -1 the app expects. + + [select] applies --limit and --path here as it does to the other phases. Without + it a scoped trial run -- the first thing anyone sensibly tries -- silently + rewrote every chunked item in the database. + """ + items = select([item for item in load_items(connection, "1 = 1") if item.continuations]) if args.renumber_scope == "retyped": items = [item for item in items if item.base_path in retyped_paths] broken = [item for item in items if item.first_suffix != 1] @@ -549,12 +600,14 @@ def phase_renumber(connection, args, write, retyped_paths: set[str]) -> tuple[in f"{', '.join('-' + str(n) for n in starts)} instead of -1") else: print(f"[2/3] renumber all {len(items)} chunked items already start at -1") - problems: list[str] = [] + errors: list[str] = [] + notes: list[str] = [] fixed = 0 for item in broken: note = renumber_item(connection, item, write) if note: - problems.append(f"{item.base_path}: {note}") + # A repair this phase exists to make and could not: an error, not an aside. + errors.append(f"{item.base_path}: {note}") else: fixed += 1 if write: @@ -562,13 +615,13 @@ def phase_renumber(connection, args, write, retyped_paths: set[str]) -> tuple[in for item in items: if item.base_bytes != CHUNK_BYTES: - problems.append( + notes.append( f"{item.base_path}: chunked but its base row is {item.base_bytes:,} bytes, not " f"{CHUNK_BYTES:,} -- the app detects chunking by that exact length, so it will not reassemble" ) if fixed: - print(f" renumbered from -1: {fixed}") - return fixed, problems + print(f" {'renumbered' if write else 'would renumber'} to start at -1: {fixed}") + return fixed, errors, notes def verify_retype(connection, retyped_paths: set[str], mov_type: str) -> list[str]: @@ -629,11 +682,23 @@ def main() -> int: connection = sqlite3.connect(args.database) connection.execute("PRAGMA foreign_keys = ON") - dictionary_row = connection.execute("SELECT data FROM CompressionDictionary WHERE id = 1").fetchone() - if dictionary_row is None or not dictionary_row[0]: - print("error: this database has no CompressionDictionary row to migrate against", file=sys.stderr) - return 2 - dictionary = dictionary_row[0] + # Only the phases that decode or encode need the dictionary. renumber only moves + # paths, so requiring one there refused to run on exactly the old databases whose + # numbering most needs repairing. + dictionary = b"" + if any(phase in ("retype", "migrate") for phase in args.phase_list): + if not table_exists(connection, "CompressionDictionary"): + print( + "error: this database has no CompressionDictionary table, so there is nothing to " + "migrate against; --phases renumber works without one", + file=sys.stderr, + ) + return 2 + dictionary_row = connection.execute("SELECT data FROM CompressionDictionary WHERE id = 1").fetchone() + if dictionary_row is None or not dictionary_row[0]: + print("error: this database has no CompressionDictionary row to migrate against", file=sys.stderr) + return 2 + dictionary = dictionary_row[0] def select(items: list[Item]) -> list[Item]: if args.path: @@ -641,40 +706,47 @@ def select(items: list[Item]) -> list[Item]: return items[: args.limit] if args.limit else items print(f"database {args.database}") - print(f"dictionary {human(len(dictionary))}") + print(f"dictionary {human(len(dictionary)) if dictionary else 'not needed for these phases'}") print(f"phases {' -> '.join(args.phase_list)}") print(f"workers {args.workers} quality {args.quality} window {args.window}") print(f"mode {'WRITING' if write else 'dry run (pass --yes to write)'}") print() - problems: list[str] = [] + errors: list[str] = [] + notes: list[str] = [] retyped_paths: set[str] = set() started = time.time() with futures.ProcessPoolExecutor(args.workers, initializer=_init_worker, initargs=(dictionary,)) as pool: if "retype" in args.phase_list: candidates = select([ - item for item in load_items(connection, "CT.value LIKE 'text%'") + item for item in load_items(connection, "(CT.value = 'text' OR CT.value LIKE 'text/%')") if item.base_path.lower().endswith(BINARY_EXTENSIONS) ]) - retyped_paths, notes = phase_retype(connection, pool, args, write, candidates) - problems += notes + retyped_paths, phase_errors, phase_notes = phase_retype(connection, pool, args, write, candidates) + errors += phase_errors + notes += phase_notes if write: - problems += verify_retype(connection, retyped_paths, args.mov_type) + # A verification failure means the bytes and their declared type disagree + # after we wrote them -- the most serious thing this script can report. + errors += verify_retype(connection, retyped_paths, args.mov_type) print() if "renumber" in args.phase_list: - _, notes = phase_renumber(connection, args, write, retyped_paths) - problems += notes + _, phase_errors, phase_notes = phase_renumber(connection, args, write, retyped_paths, select) + errors += phase_errors + notes += phase_notes print() if "migrate" not in args.phase_list: connection.commit() if write else connection.rollback() connection.close() sys.stdout.flush() - for note in problems[:30]: - print(f" note: {note}", file=sys.stderr) - return 0 + report(errors, notes) + # Non-zero when something the run set out to do did not happen: this path + # used to return 0 whatever it had just printed, so a wrapper script or CI + # step could not tell a clean repair from a failed one. + return 1 if errors else 0 items = select(load_items(connection, "CT.compression = 'brotli'")) chunked = [item for item in items if item.continuations] @@ -686,7 +758,9 @@ def select(items: list[Item]) -> list[Item]: counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} before_total = after_total = 0 inserted_total = deleted_total = 0 - errors: list[Result] = [] + # Not named `errors`: that name already holds this run's phase 1 and 2 failures, + # and reusing it here would discard them. + failed_items: list[Result] = [] mislabelled: list[Result] = [] for offset in range(0, len(items), args.batch): @@ -708,7 +782,7 @@ def select(items: list[Item]) -> list[Item]: if result.sniffed: mislabelled.append(result) if result.status == "error": - errors.append(result) + failed_items.append(result) elif result.status == "migrated" and write: inserted, deleted = write_item(connection, item, result.slices, renumber=False) inserted_total += inserted @@ -749,15 +823,12 @@ def select(items: list[Item]) -> list[Item]: if len(mislabelled) > 20: print(f" ... and {len(mislabelled) - 20} more") - for result in errors[:20]: + for result in failed_items[:20]: print(f" error: {result.base_path}: {result.detail}", file=sys.stderr) - if len(errors) > 20: - print(f" ... and {len(errors) - 20} more", file=sys.stderr) + if len(failed_items) > 20: + print(f" ... and {len(failed_items) - 20} more", file=sys.stderr) sys.stdout.flush() - for note in problems[:30]: - print(f" note: {note}", file=sys.stderr) - if len(problems) > 30: - print(f" ... and {len(problems) - 30} more notes", file=sys.stderr) + report(errors, notes) if write: connection.commit() @@ -767,7 +838,7 @@ def select(items: list[Item]) -> list[Item]: print("\nNothing written. Re-run with --yes on a copy to apply.") connection.close() - return 1 if errors else 0 + return 1 if errors or failed_items else 0 if __name__ == "__main__": From 0348b4fd4f3aeb6ca79fbadd4190ff3a429ce511 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Sat, 22 Aug 2026 00:30:52 -0700 Subject: [PATCH 06/16] ADFA-5153: Keep Python bytecode out of Spotless and out of git Importing the migration script -- which a test or a future module next to it does -- leaves a __pycache__/*.pyc, and spotlessShell targets scripts/**/*. It then fails the whole task on a binary file it cannot process, which fails the pre-push hook with an error that names formatting rather than the real cause. It blocked my own push. Co-Authored-By: Claude Opus 5 --- .gitignore | 4 ++++ build.gradle.kts | 4 ++++ ...content_to_dictionary_brotli.cpython-314.pyc | Bin 51192 -> 0 bytes 3 files changed, 8 insertions(+) delete mode 100644 scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc diff --git a/.gitignore b/.gitignore index af44d5bf1c..c19a6e0552 100755 --- a/.gitignore +++ b/.gitignore @@ -193,3 +193,7 @@ NATIVE_*.md TEST_*.md assets-*.zip dynamic_libs/*.aar.br + +# Python bytecode from scripts/ +__pycache__/ +*.pyc diff --git a/build.gradle.kts b/build.gradle.kts index 9c3dd7ee92..70b1567f55 100755 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -338,6 +338,10 @@ spotless { // and every .py already here is space-indented. Only the ratchet has been hiding // that mismatch: an edit to one of them would silently convert the whole file. "**/*.py", + // Python bytecode: binary, generated, and Spotless fails the whole task (and so the + // pre-push hook) on one stray file rather than skipping it. + "**/__pycache__/**", + "**/*.pyc", ) } } diff --git a/scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc b/scripts/docdb/__pycache__/migrate_content_to_dictionary_brotli.cpython-314.pyc deleted file mode 100644 index cd20b49529d24d4708419767af2e6c7a8c19516b..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 51192 zcmce#GcU0fsP)Bm z`{tYXJNH%<3M902l6fVuP|K~moO|xs?m721<>fjA9M`&Ty!83k1mUmgMm;S2!tf|+ z6od&u65_&nK{6aQ95Kd?Rdj7UYKoiKuQ_gJzm_A`xK(*)i`&>Ud)&@`bK*Jd*AaKH zUuWFOeqC`F`^}B#;@5OE?}$6@KH`acssw3IJYO=G3X-K%81cjl&e?nF)#{~xs|Crr z*}(7PMbh4Qv9vE72@ zbs@W6D;43om|d@vN^o7uuB*m?D_)$F=fs=@Voc3mgc;<}Dq*Gtby^|))GyAeaYL2AUEm%VG0HsHF6 zU3;ZwT(_|64U!Mnt?ar<+AnRy-6p!j_svop?%LVA7HKoCx3FuUv=!Ie*mbM49oIY9 z^+stYu6MEPP3PLA-P_L!La^K-1iz6-nI4~RtkHzAYUW{D|$jCbw9;9rLzt7?5y(EW1 zV)RlV7K$}fQSnRRm>7|RA-P$+7#a+W$3kK#FnCFfjD^I(OM$WBP)xjbDZsNF3%wj? zZ(^ZPY^TE^ikp0>R{VN2B#L573$H|+k`Imq!lPn5f?UNoeRgp?E?$a^1jT@O_-JQq zPxs;0V?7;*TThLL2QT-AM?=VM#Ssl$ABhBlV&ncpo%>t1w6$&W`o!K#AwVw}LIKn~ zRyNvjfeLAjqSUbq&8TrCmL)?4R=ucijOr@J0@v~BMFvTH7K;$%u8Y?~GN2R*kHtp- z?UC!NtETn@YOo0)a$xW!@B%u(fuXPrRNo|?2p^lJz@h#h=vOsM<-*a-DnU&;?K`z>SC=qb-Ep4Kc@s6}?7? z-M~jNqi|w7dSGQDt!rbUkr1PdVk|rw#TduHi^hOsA!AsLb8e%luL~*~!GQh`- zK{4cyAV1VlL06x6nvqy>d<HY?@ef)95*2ke#6A%|LG(Glv}UbGB^ZCH$r1C<#hsKp>_0Wl6< zgnEsGRRuv@<;Zo`yHqYwWfoBDMm zG=%nyr~|$xz>89eVz36;0+kp!9VN1Q4blkx4udO%$3hCz#FH3KS&Zczt* zQ0-{25t-zQ(5M2cj+X&98bgE#qmXn!gZcQ}gz-6u&ItqgL$O^V#(Q{Bym%cq1XV!P z2S8KrE7Vs<9T@E)il7P!fx-AV1AZ8lBa8!Rv_xY8^l5B-7}88!se6EdpI)>oT?;ZB z3k-u85$PhNlta;o46M@AZP4WL7@o0lM4cgv7ks0Us~1>yLtuzNW&r=f5UA9Z_ytju zZ+NYao<7-f;6!I9qrojL__Bq_8Aj*eWf1F7{8|X34B%t=asDn|xC&TATCb3(MCf@z zY=oEx>VvvLNQtPVfiW{NJR}A##?S^Y!$CDKC6m$U<_pLlsDAL0jQ$uOrKXOAFNa3L z#GAk=R7ip&kcAya4!%wZ7k;#@kQLbh}{sg1%6;SK(64> zkBlKd;!srUm*+5!EHD!GRt&5K}0 zLttpAabQr6Kt~va7zD!XI0K{(?PTb*o}d#&s-1jj5L1flR04(Uak-!2D-)}pW!_7cW8>k4AaL(k)a@>1Z0l?;=$ub#m2pRpnPrg68w7h z_a4~^nkV*L2fG;~b&_Ff=;bJQ0WblriHth5euB)x$PbtXiI9X%3KP2s5gLjk@d3uR znCQ#6S<4}x!)r)617o1AOdp7KdyT0a0uWTOSS)41@Er*blF>pjTnJ`dn{fL`#KnZ* z7tR~Ee*sD~o>dB`43`X&K{84v$=nJR*=zX)nHy4;tAUa6P|5_FlClCKU?(Z_#Ykl2 zmVgRaF&;+)@fa0RlNxNj6afJbqF;io`^SQEBpf^)9}k8j5YJu$qQ_bfqKPNRdM||z zM_Rcs7Hb7^1~0ZgBPyp{0|QE{LjwacG<31k`4ycte`xyG5^0E$C2hOu&P6u7{ZGV12W8v3HfdjWC-lwtl2 z$TvL&nQ9Ha9DuAGy0A+d9EzfIk?}lAcEJ$|28o`b0@p5a*$JNw#(8e^n9I;iG#QS8 z^ef8y1xSa)r{!@_Kd$9D0?{bg$3ZT85qmoV+D=4TwFzv7O~8>dQ^^Snom+tm@; z6GA_3`O!n8aK!?S>Pi&Q7!lll1ViK?lKG*#*O2q0nI=b zHTg|F)XEiyUr-SprZY4B+l+eBfuUiHS^5bM?C7DfxZR7WvY@y(CJ4U&=Om8M4P{G-Zs$K&XOH5XVv$W_3t;1`ZuP*n9M3_kQX50V3EbD}>&c z!|{||4h=(?3CSt<0MurP-Y@9%cizYZ(hm~ z7jwj!gUfd3YdJS^rZz0v*M4Xc9M$iaRwT_uAJ_$p^NS}xck-(zXL{c?Z}>15ukg*s zF~Z6p6zp%e{J7nDz?AbZA^dC*$ka8molb-m@)|ll4fS6^A`SJ;hTMk&PSfij(9cXQ zyIYO=1U!cNH0~#ail-uf21*KqNNLo!3PH>&5ZzvjL9$*53mlDZ8M?hDxe6r^7o%=X z*-oDBkmZPsu}p2%(ImMJ_w2+PRLko1z-Y4M-nTnm%Xu|ts`l61|8(D3@PS~kYfi6e*5EZ|bOMb@)=Zw=poRb2tj*-iTJoZ8)XO zLGY;fY(?$?~R|AWn1J?rvUm_DmdFcDi5N zcVpjlVBT3ZQ}peMw<>0X-@Sb6@{;eldEaw)O7D6Xe7#G)7v_C0Ecp79&iNm5kc>X_V*zQBJMb|e zc}VWWh1bZyDc9gyK8(`}D3&NtB+*i*_{!Z#${MjA7ZZX4&&B$!x6R42J+}eX!%u-K zQ5J@`)O}CVl|Ze{l2x*`LZb27SKx2TrkE2_CfH+A)@x7&BiA@XZAZpW?TN1-QOwJ# z@FAF}l~|~idICRl2ieb^LiUq9dV#&<2!g6>2hG2;rucD@iPZgCI=?{`^CW}rsix&f z#x##5713u;_aPc?Nf=wM;P0Az0)?=?!`%?55)B6~Mo=f>0UO*zd}IkHg-R*9Knd4@ z2UhlHFH$m4wBiE&=`2{z$^F&X)V43ZGTr;F_L-~S+%;!j+}e59d^ec1pI)}RA6ZN; zD|na1`Y#+Y-mEl?$WhGT6WMtLAO{Q{S`(y zS4^8J59N?*ci@Wa>@<}P#nqJY-pnS(kwYb%8rXYl#(SIYy`8H0>@^1fR4eGTfk9z&--aRBBJ@x;*+9enzn*#nr>9ap26 z7`qCi5eOa4utU(0#fAnlDvytm2g2|e%#A_c&hB3j`Rz{RXw!6fL`1ARM&}!t;LmlG zJJk2lv0=QY(cjHR;Qw0wUmrX8r!PfA^v=xQ(OEj$*|}3D;rtgQ;C3?_+}ClSC!u_C zEp#!uvzw8=?iiu(ffH&u7e}KkIT@i6E?&UPEn%`zQ`U7ohdL8(SQWToI5O4>WA0A* zMZ97K^0LTw&yrt>i{i)b71sBUbNB;T(pjQwCp+5`jv?46TCYWhhT7DJorxUnA$xYR zxA$bi&Ynf$L&{Y=KEz%Mf!Jumqp#UDJduk~Gp{j5u!h^j_H6;}a!0^x;$LFevI|U2 zg!n~#a7d-}ME!J{2Bf#K@X(MvfX9ql`~xl~g!gTElk&g$RI;Rb(boK)yL?%_->_)g zkiMUaEZXYcb5|&@r-m18YiAl}Yk#|CZrftbPTiZ(qOBtRU9!ZxX!E}3UaSA`J-0}= zv48W*)U`$1y7z0FW)J=LiMhVT+TBUFn6#g`(}Yvz7a1j|^Jf)#&t0Y@PL}LhwCzdX zPrbBg6W?=Jzvs+NlfQqKIFp??f9zzs5>tboE-C(idSo2~`-g(Ta2^^Eok>=t-UPMHcAX4(sd;$RqOGtioFj5MSnqGvO0TBDT_>=Ln+e;3?R;fN6vJ2 z4IFsBx8pR#E5gT=AjF;{Wsg$+PE@=5rt2bN0r_-S6wA`;F#7 zTxbWHK!@ScyEsn>=Y+UH65_`5|0!;&MZ@A|$#~8pnd1~ zG1()PLrDk71SOwB)l)Wh9}cFRBk1gLOfn7N#;r{Z!MI8foMaIjfvpQSjGCof`Xni* zZZ_qXA>~%4$p(gCGLFdC6Xk4ZafJ*bEqSf+)46l0T*^tUYb@npHDaSM!w{kfIEF`+ zsQ2SU>sH|NEfW`~bC8ZTQYE6qhE|$hQ~3m8pMG-S|E6SQvLhYPU^zx38|Udk{=0fa z6M|wy=R6`K2Gxwj)RjkiY)e8Zr7^EeAqXDLtg1@Y`tb~eFffRyA5sqOGz2JB4aW0? z&?|I8ZYMRf3W}!R__8ozsS>)-a-src+1g?X8)W)`L&cOb`Zmf(ai21baj9ChN9OE? z3jt=IycaAW;e|Se*4tI z)*~M=p7yzdA{e;jaVpOQw-`B&DDV?N$^k-BUsRrPB?!wnTbK^7L zDx9x9Iu~5pdgR{LBMY@h7pjlG<35_SA5C|d$}FB4mE>zE;YsKtzl;R;s4i)Q>0k)c zp&(3$AzePwu|p|8=h1oGJ`Q2Xx<*G3<_MBovmw=aK`-Z?H|X!y3+FKyU|ArtdCR~c zEl=P&1I*0*Za}ex^Rb|RG*9WdF8h8{>r&Gm{7*LRnT#*FtLNR-Gu7|7YiCZq<8DaW z58c^>Q@XLA7}&rID~e-kZi3q_K_#Bdi3$9~0Ak>CfFzCAeKbt>O?%l-x;l8z4u$>Z zJR)WPaj+*Adj|6yjLG5)ZnwEo0EP;x!WgP43YMxcPj(#^_^|r7$QModEPmLylF6Ku zIkM~=-l143mk%s96%mTPLn(=e0I3Ir0QtfU45SY1FVNf;2bm-!E4*V z0H#uLfj7E|Va$?7sSQ`?WX36FCL+%9hz(#njE>|zcU}G*rTHSAzC;O&2OtGtdUyCzN~3tJYAEW())e9rc^%~K-xi&z+rgw1)OzqM=Fcl zK@y0W`ctwfk_D4J&eBWgpCr|5K(PQOm4 zH|RtVl)pkJQeEWVqSKpn`Z}F{gH9_j<#l>SrxgBThj1cH*~Br$<^Fuva$)hr;mOkr z=HmCtt6?gC z{YC_VclMIGnD-!Mn;|J$@ck=e;n@P&qMp8PC^-s^u;kWjaJi(51z5FH}#I@Q=&TWKmI6#VB ziIazz%d6s<$6R-qYiLkMel^!jilo7eE4glwa{%}9XYQ~R0Yly`K91sBkbTO6>AQ&h zaVwv^AO{>wU%+vQ%>}_@a11s+vVe219?ehrmXVKn8tEzXVrENkXgh6T+x(}rdW6w4!@X#gj|zJu9(Y^5Lc?khHatLK}LE;!) zk?6(fIpaUfip6COG8$n1`Yn3@f8s=}3Ct$QOFcr5+vu_RGG;_#gvXevq(kwbP#hx% z*f{+_!zkK6d$wK+|8(otJepXm~zqER$_S?RX*GQd|)!H={LaM zlizRnCx{S*Ws|m_J3UjSU)noy=x0v%eRtV(?KgaH_-5;Gmn^Sup6y*~**o8|H`%-| zxo;#{J&FuG5y~JZV)(!+xJssiuaDgvo4It`{FDBq^!!4{3(ITQf2V!++P~k8w5}aU zYT3a_2Mu_YK+u3=1aBj2>Eq|}h~P;!{Mn@($ssvgt%S5(>}EhGafd;kQ@$R$Wo$#j zRmx4iv;*Oxf!HWH#L6CFi5tjQ^z4P+xDieZ6f8-cbD-M&I8iiB742jPQO#K>nO2CW&1wo zH>vhbrfhvWnx}&}=C?DrQTF@^E;;@@m(XU+fR0UTXRg{#)>EHJmk>l6wC10-UxUK^ zY^|Zv9Qtz7+%vd~)fbq^55HEf-@5&WD52?@+T&#Ka_KNt%LiO7*Kfr+ugxO4(;Voj zxZ~+B&Q_~zWuUL}(>Z$!0$;`O1$%>630M}~zC)uFprcy;$d^4MfKKB;T9^ro+d(?` zvmYdHW5iLBV?XqhAmSDf!3YB*7#R8oR5lzCP=g5b9ETZ$dpPH=qmpLzI9L^hePdN?ev&Pl`1_)Gh)Z zxT2=EI8F^5Vxd6@zGbs}#Ny!!Br<6LK=3&jHB8x*$RJqf(>(fH=xq77>GXfmiA4Sk z9!^X(+UZq@m5XyU44l+ewk?U2Agw=XFm;}R5iX5XlwKF;92S~>_=NB>l;M-gv zHsBHr7bi-o6>u2c8j z#frQ!JU5i=xVq@Rmb71kt7t>}+~%eBL-Xy2mg+n1)pvZgZaQbNpyqDdMAru{VU2Hg zXrXk=yt!!N@Xu^{Q`;5`8`4>YZ(Ec8z@qzN(th#hRa@r5ck36bPR*MuCk{*vQMED( zJ^W&_|6=mR!DL5p(H%&I_UBu<=b`q2$^jlE5x83G5P+zzU{nU-#Yg&20Ey|DEQAEhm@Ds=gINeHymkX(9>iB&D^S z)Y){=C+zrVE1eEZe~j@U){hI6xE9|*?HO8s9P zgGJVkf%&rhHTj3~e@{<;_Gqj=U1D0eiMsu6f&(4FGsJ@Z?rgs6=QXD$zj+6x-N({e zv-yZVH3wUNS~$yInx?#~aPzy`&FVEKZ>|f1PEEG^ZT=j;!|(Lx`t$rS7i|M1=;%9J zk8{@)hne3BIDz6^GsBJC6SYP2qb|RtM-%zjC+y_+sAWY9`~^ekvr3_>3JQ5aPeD4B zdX*IhHh(_qto1wU42V(Bp+rsGP;*ExM1mTMs9mAIFl$Mr=`YkP$%Mr`%&L(M<<#(U z`peW5PcD&`Yr*#lZfjHDCQN->6MxSFz=#f??l|6Yuva|j3nK*ippRxt=~ta0J$i`l zl=)8zXW2tFaq)!jDyULzw;m@qV54qeZv5*5d)1nHgADEzKG;K|yI zXE5PWtMT&|estB6{|oNr|BBN~x`H?V{@&l&%L?}9rfi{?LxY&mNZE%%h{b{)oN^sJ z+1-1z`^vs$%0|IZ^oTkzWrgF$ zFk(#{>?tdZ_vfF{2dv*KD4hi{{*WdaET}^>BoYOSxnSZ*_m%t?NV%V&M4*}wmYu~* z&b9N-wGWJzT+aiCkY7IeDKN{5oCi*8j_Z-h?y`R95p3?yUi$n?5Awme9~KE>?SiZJ zjrg=_D*rYAr2mchL`Tw9yS%n)qGNf>wuuve-G5=;es1b<33v%VGna*Rpd(^#}Pm6?u~#U+Q}3 z7S0(0WJ>n!nzwr=a}aH_b!O|br}*`pn>o|o1yAjyW!dRjbXG3A@|Rpy^RB8Tc(b_b zXRUKq2ty6~-gQ0q(1yPMcP-h==k4W7_NsaOuicrnSG{ZB_0S~b zSCK8KV5;u*=9|r5@y*xoUvwTwnh$*R%RNGF9dn17&F`o;|Et)%IkNpnLI~=AX8h(B zodDf}FVip~0e}sY8iohyptVs#01#my067}BkS}oylPvH%*Gn$OEw?YTr>n&Z8rNVq zhDXO32Kfyne6KtZ_F>2t%m(4X2=ReW(IO|Z8bsmAbfpcGw6L-$leCzcr4g?lQL`lU zXf_`8DJJL)I-7^ae4BMCH704v(6I?XD-Ox%w?GgwwVC{8)$#vH;UikkR==5ArTBIs z=PUJoqkc*sEmK4y_;bODmw9fPqG)57U*joQnK@->P8s$kS`&<+@187qk6I;~D2G6c%J{3rDAPkAavqZr04)6=6k!C5^ROS!-U2Sa4phHi{- zVp4ZjTWS%Z0;137SpF8@GCZOw79u3>*LL67J?Wc^FKs(|Z`;wuievNJj=uHE%+=W| zbJbt##Lc~mWA8YQO&nYv0cIAmHbjYA4A1qpYif7R19gE)=eL#)yt3hjoq z59ze*sd-?+@B5Ct2NwLou${Z+fsL;0g1hiR4qZ8f67hkPu3Un(_+c)t`1sb17z{yW zMynCffs|k#0`@}a&N|Ld7<$c73+$IjyYj$@GG@BE^qtD)KLrs~5$zCUYSKh~s%P zA}d5KetWw?2d!KvU+5k6h~{=xa~cZ8%jvfvw-{(Fpez3$erHxSB1h8lL#g(m4pztP zrCR~FEXoC_CX^f4Z%uellE{jq?{Dy&9IaLy?OK=~`rWmnC9bXkC^OmXVMLC0J=SzV z)Mbjx|1YrZ`!fmyV^rK6br3dydfO9+``8j^ij5U>{bN|ANLkg176Q}75R!aI%Ym>} zWMGY@Ti8;u97@pZ>rU)F({pHlZ-=TYK(Xo7IKduqpT-ur5(CX)pQ!N!-0YP9Ga%jF zO%gl{#!rZcpc)~=4An&cN#ZM-ZlUTwN4rmVNWI`Oy(hJbG;*CvuTd!+YDJx}V3np; zc~@oa-+%l}$7!(ldyF>!J-uT2zDHO84^D{I3SjCsC^NoD@-=6Uc7%yzQMR$+lwIww zluhX>HmMn-!MwsAs7T!`_?%B_{t}58+5UCVZWseSLc`X3^?N3ROL;Z-@@nqvn>f5| z&Ra5<%$rM=?MIe#+*5`xmrWi1M*kcAN&AKmEC_A-&><9*-`GDLU#i+TU$t?f<9=@b zSH07nGi_f#HXB+j*u0Pn|C#0OyO*{fySM$=JMOOe?Z@WVZ=TsYw`tn?>v_1FIC9V3 zwOm%7G?z~t`OqcUJyVC4it6Tz>Sps7i#9Bmtb09jGxBap!(`{Ot7OTwZr-&nS+)O8 z=baskt`o50OCGh36M4gbSp%FTONCRElus1*t;UT?kG`fkB` zlx#2HB6_l7&mHUS>x=fIh!C6F^kvtpHdMT@7WHryC7qQ?bLB@L95x7^;|3Pj@#9^2 zow@c5@mv#i8_|DEVSdaSD#{G4oy7&_JYFQxBOqc(fO8z-cidfF(A2_9P^0kWws9o|uA_+qo zG6cbY;$JM6rwNl|_wD}M-FK^#_T$SrYoPf~x8sC?Xt93u$UEpWhOt^v?-kn)KAlMzehty)pIQ*DdUt}+lm2l67<}(M3zlvQzbhxolC}J2DomR5+~Hf62_3HmuIpAQrPbPei9L$SxQTOkJ{>VI8laU;6w!e28j?Z-mPZ{4 zHHiDW4~bMX2@hCHieaB(8-rqk$wr2UAELSP-{8a@p|y}9mdOBPjWPLSyk~&sdQ_TL z+YjHZy<7UoWX!exoi&dckSv150mk9VpEh#}%)w58Re`I3-S9}n!za;PzHZe@T1(_2 zQBI04EuR57ge{eU@z0mYc)E7=L&BL#R(3iFJGpKnkGtM-QWtlvRvC!}ja3BMr;ouF zEj|pohJjf^xUS*gmJqhcK(*i0ZiK$!Qmi2$V>+jGs==N5GiKnMdp@HM1|877TyESNBK)l8 zwuFSBca}%ZI0J{Xa$6`oR5j(G6`8R9$xh5$;TuGAgHWX1J9G|CS@X(Xf{xD`vaOz? zD#UuOt#VTJUDS-cb@f#1>o+r%7Bf?*t1){TZcamFbk@=_1T1H)<0dwfh^}F#K!iPA z31JQqC9*k0ThxNt!^eEzc}^KNmslDMn3V@=VD9_89?qnuMsqMcvnb(`#Sb%y3W03u zvPR4uen*2+*f8s_t|L%aVHC(PKG=2unUwcu@g7RDWNm|~unKOcOkp*uxdbs^$c!w6 zq$>`n@jyuCV09KP&0hD%7&TBL)Ox0zTiSJNa&K7nWb;yr4yDb z(wa4D0a43Ibf3NyrnPgqs}iI&lFYO?L#$AVWxZp{TJcP~rFA@8uok|$LQjC4UDVil zA9J2b)MRFY*m~v>!=4V0N3gtcVLQpqxdn%?JmoMPlHfOl8j;U01T=jaiN-84q?&1lN}tGWQB8@^MO(Z%JZ$GH zsH02_hr7=(iSkQydznsuLZ^>#f{#sFtR$KUy8r<>1?`qrBvsYkqx8xI0JGqylobp4 zVb&{Up*1EiF_sbE)+e#;C^A^$g<7(#ZQ0s z)6@1vo4B0cK6mK%PrQ9%Zr`1jq`N0+?}0+s(6m&){a*d{MaPc$`t3JIriP{iGx@J> z#m&UQdB=|BHI++iHY~2$aKklenA)^lv__|h{`UD~?dD|u7U+(iaz0g+Tz~AY|L)NR zkAKNCFz*>y@B}6;KXZC+Y?(SUz3Dg3&YXU?xN*6pa_Vey?dfD;?>$%V!yKWkiFE2* z)TgF?sk(i!x*biQ?3~`ODZ0-xEljNz-O@#7zX|lxLK8u4=JX;)*^Jh3 zAw_OQ!^dE)h46-#_eC#c)%g}grIi(kZjr0XL2i0Z>m#Ho8=SK=VgVu(h6se zL|2bY`Y&6M`x-5;eN6^oVl;|9gI-1cn^j`IL+@0zC)aJB>%6_?cEg?gWWmAzw*IFZ z{&>TZ>&&AZp@PR7W(+Jd>H|dlWD3kH1XjxsChFtdG(_n9R9W=_V4y8IWHD5bCR;Iy z#Ygco6=Qd41q&InL#yAf;}$;mb_|5w7zl3#$bx5@z=L~Z^BScV_=yT#dleyse{E}nT^*}>8V;mP zhr9&x3GDeumC$WS_V-w z?RAi`+=b=R_-J5Eu0~o0y?zFSKlf~yG~WjbT=~-0sYAcIXZp}g$D1dXy)AQv->H}i zOq-|9PM?~NPY)#v>+ZP@-#&Fae*4Is1Gignvuw|q?EP}is|L6W)Imb5yf z3e-NqC(we7I+CdA%25kuv#f9IyWM{Q5 zVzoB*h^tdcIV=Tm7|N;x-(-bf4{&JZAvxhzsxW$~Uf;Vtz?GGU*0cF=t@M!c7kq)$ zDqm{Yfy6;hM?Df-T5b5a@-COv4m}E}dwNujP?x!sr}m!uau^~LQhzd7yZ!mV5fA!4 z3nYOfTGS9mY)-ab?E9(TgLzklk|9-A9PxV|k0X8$WF?{RyFHD%<|&sMGue*5AN6c_ zN*ZT}q3cJA=!Mb34}T5}3k7F9ieUlVixlT7AzDDV;46oCKkU(kJlt5N7qEl z(rs1@7+_caGO2*!D)=_2c)QxPJ{NFGMSk@mex*_@+Y7IUl?ROl{~BqH8dJu~ERR-SWx0G^;cv(DebxdxKRe|K*?)V3J%IP#@m{%dypBZU^3aIev$rFD_kY?r?JP{ zvnQQGy;4(3WpG3;R|S8-v5YGfaMiBV^<@QUQYCm`k-t(}n+sJ!#VOPfQ_F;Fy436X z;dadrze7!l9)W(=L5$qApAAL` z`cfIw=(AdX2}-Z?my!~seoqtuJbYa^>5>gs!$I6O1= z7^5!5_ss;@6=>H;1tX9*@z*fs(1Zn>sbfK5ut7X_PeY2=p`{Vo{D~6$tb|1)W=*vV zHS$?9Y{o?{xy?Kh3%^Wz#pZ?`6sHaC2eu)Krb65_g3$QxVr-`@+EUEHH&0TmJW!B= zkzG6k_8rz9fC|NT=-@G8O6V1B)W&9Su+B4AzkH5F!Hb((+s`Q*k8rmOHour?)CZt5 z5U>DoEEvZ3sze|Y-_?m}V{$I~PX0D9DBd{5Rb7^qI{mN3_cmmh@M2Q zCaioJ#Y7JFF2OWVQ1m5=*!Dj(#iWLmV^%8hToykNHklwcuAXx6iaOVh2DT@hmz2e^ zP@;@C@L+m3iI^xhVe=tTu0;J4g<_+A?t;GYCaSZ2#J30I-Wa0%_h=HNefnW89ZVI& z4LgW&15pi+$opsl3JF9Oop|Jhk|O^hUMY%J%6R$eO1JXVI@aab9e@yWwX6GcC?iH9 z=7ey^cxTgpv3&n36QJck!n2R9ya6d=TU)}ocXJ|_-}CTzF_A|Ic7W}GDYont6K=ZK zg4M*vl%a{)FPW1kI*1(%Ayj7+G2z5R#UVxf#s9*AHONgbMTZf?kF`Ak8XuuY15RF1 zE}-iWY6e{#{_OvguCS%TMKauc2RHKH(i7XzI4v6huTX@C%Z zGc)7Y=_*O5H|S$K(2GqPAw7#6g<2oMrk*nBjD8)H1-c~@Whw{eMQrtU4G!?Xq=#Rl zhkT?Bu(gY5N;_pmMW~ufl%$%PvzA#ku`>hK!Iz2C%GGqD1St#k7B+$ygd=M_mNqod z0_^lEm~!zIaj=O}JTQX2b!c52e#2avVEVpfjV^VcZ(p)of{4WQB3wfdWfnc)l#n(P{<5O?$ zo-O)L;`WAx)~*Hj@w>r$ZV4|Z4kz{?Z?9dpmmbtU(~$@%%E-oSisAbI*?vTShC z6@1_Bel71t-n1FUtkUL1dkaDtzft~1`Ap~R{)IB%T>d*{ZIjl?t2gqN?M2^kz2SPt zUOVYpwimwsshgjA$6oU#=ff7EaN`FX1$Xh=rHjQqZxLf9C4#&bxKAm8fg}hS}hyOf(4Kr*n7gsG6H_R6|EPJHnnHQ41{g}5e$(!uNOz+zDOO?&@ z=$D~n#g@sAsk$5R)y_{AwagAJ`S#BH_9k2Q-M(_C_>L`k_=RM_i|@Mn->+!UCEPL> zn(IsMKA9}&dDnI7VWCh^_Mk+lZ%CT!C%UK3{draWyuD)5Jmp`ms-5wF{nPXI%1PT) z#n0SDtm{jr(NE%iaRbcrh!1IgtNC{Mo%kPp=Et9z-{DVQihN))R_BqW{m9fXs+(Ie zed!%n{UoAJ%clF@an-(h0@1O#Yo;Bu4ez?z9##lq!-tiy(=$)8vUz*iqP^nhWmPlQ zZ@b=deKT)b92t* zi8t$JTnptb$ui&U*bmR#*|o6eWO8>8VrWho@_`W>ASy%#l)r=!#tIE3&{XcQ}7cE|^o-In2XJobqF{@GyCyR$20cDDTPA zYDBfFFyhg+X#<;N(nY`8R5(GgFzcRMba$?^EGSuYz?A3GTDqR~W=^&kS;Vns=~J}Y z>36L#!l9e}FfL^%x6ZCZ9LoN*Ux2E>W-$Y*Fd;(>l+kXgKWdQ0G}y06y=1ai@B+{L zj!)DIibc+nyOaxMH!ocVyLY!~y$iLsM`@Q+gCficP&;!ue4%!FbZzFUZ9Y`n0>G}& z@61$1`@ZjY=u}Dg4JbW=Jw+1KI{5j<+)!W-zb~f?{j=?*#zf>P4Yf~2D)e)*F`xXNYKPI3$&5M zniZSVYd#kr8{uxI(N^>$O&P!>WO&qx3vFRz;B6FBMLH_1#p&zK$I6n&3Z)~z5L)sW z>dQYMGlViQ*H7Gj3b$--M?b40e;Q9y)v7$q+o03c)D@A>5>KI_z}7l0-~M*V?EdNY zZxzm*`ew;Dc7JX6@0KX*9amKwpqxQ~SGTim*_0?*4IZ*8zrr= z9jAI(`LN>gtV>iJZK`z{i`dgbr`S33`q97T{5?7A$GLb=k9%-3K z@fWz55FR49pzvES%sFl!UaUK`Q263}-J$8;r6VuAbL55B4^17y?R?>j3$A{0TESX# z<@2`k>F}bh;k}xM1y9#obP?NDBAh<*4fQVP(O<6-`+Z3v~wYM*F@*f zY}jxPB47>Q;${Z15#Kv_+w!}|=5{Cd^)2TYzg~8;Y@U!7Yuhd4Z%=x5++GJ6(tXtM zkAflR7_Eeolb7_=W9gGE-%QMmB-?tEd1n^wew{++%AM%=b598-JKdGjFU*cFxVKE2 zvFP3Vt97tAa+>#+W|Fx$WsvX}y?9I#XI=zaz zvs_sE`rey+lLa-&n&<8m{ZZAAtCFXr;w9t{hsi1w~RgH z@-C{V9b|dJz2#8>&J&nKjhb|nHdEd;D^sOzNO^~{$-90tra&QYFj>bD*Ilev=ExHO_cKSCVC9mPQL?Zp4M376)FeIy3TVQ(ZjRdfZZIc@TDtGe z+Kygohpm@S z-ZNI^>M$c?azkXY0cAMZbe| zOO|7=gR^f&^N8G~XhC_17g*oqExb?JDDDvx6nue`Hu~;w=wv`zW+_iQk#bg@i)Hk} zL)R4(yF5VYh$PY~GH~oFuben?f8);EJMUau+<0;!uV;Sa$yxJjzjEVOrsMbA>v2am z$DaG1lBrKGdFsH6+$Ch6oZh_P78z%Hb@#HX=(UqKPLc=DOn0)febKdf9&)HOuX*>}y^XA{XCXOqvLUwZy>68|q=PM#e}R*o*Z#=s_*oE7uVis`G1 z&N_XqPRupVl_z&e=`ctsn5+yfx`x;{Yq3hpJ?C01WZUvq%l9__&~iJF;VqkdvHzX@ z{R`XAEwrCssyVOAIa#~scJN1|-ygj_aQ9@gvVX~S?qQXXU;jWv805sskAAscaG&CV zTQ`e4&6#T?pc^vX(arTCE z+Cg&$Aftvy{}pOypdP+Q+N*2~48|Ui6@KVa5?^0&NP;?x8>J}ooE5KeD;e09wFChi zAlunfvDL`2DFABu#B_Fqh~_*)IxO#-l>xeV{#b?!qk;~}^{gpDEOW7zcHUF#=$2xV zN6ME9wkc42Y^w{UxKtz+>r?h-hfhh`k1D-#Ef@*)EIYEXk!sD{vzncr*MbLia_9RQU{Ftkr!XNb8>Aiz-Q!s-K}pYIHf@mDWE)&b7d^x)pc^ zA#CNbS$PntKV6^Aq?3kw5Ise^jv<|9L`vtTiW3{4nQ=nx|ARHDnvjST$K0#Tc87+0&#;Pp5@bdcB%=HV-Pv zt5UTp;XHh4`vFsa^yqCZ%Cl@u7( z9}|M8BipC0xs*!C3Td@ZJ3-f0^efaQ?0MC4qJnx(@|`HH7Nlp$Nv)HbTX2%~#TK=0 zaQkmn1yME4F#O;$`fKASgYRSDvq`N>&o;I6^sm}-C{WfAjOtK2teIhBkF8z%C#>Cy zd=Zep8OHSbc7wE~R2Z?3V1Y>;u5Qha4Nxj{Dw;0+Horx!X%G_?fFsRZu%myMdX*0F zS78F9?)Jy#xkK8SZM7NvjW_UNTY2Ek9=v&O<#Sr7UT@CU zL1C}oph1Bvjr-8T=N?xlwfCz?RcF4;{Rh?3)OS%w{~`5Rk9MU#5ri~%#G3D*IQ-~2 zqF$wc*$B@9fqp3P{6JWqLE5iM2D=Oc*rRgXG@$X{s7pGK9s1PS!CC6}UTloi5A{pQ z(JUQQ#~CPUt}@BC6DEUPe{K&ZDOMgmr&c~mFE-k9diA^hUiF!Z0csB@oMAg?bS`K# zTCPxk%~v~`*Wagp-FN(Sec!sxWVXgZ-fZ%Bqn^IkGNxqgShm%Z@gzsrm&1gZn-==Q^mW*{jODnRL z-SL>R!L5QXGg_6#cJN5hE^`m*jVlhdH7DG7A6W4cx9WRl3>kdVdmY6Ku1?o8y!u%> z@BXm*w(o|DktorwVhazn#tLJG^TIe{R-`uir}h+!U|EUTooYTR=F#H2IxL5la^)|2 zUQ)|V|Egsw7)n#ggrShA2t%WV(cE-@NQZUwQ}aiboS0-gqI*-Q)&W%OsNTS&%juYo zCXcGMk-ESIP-_)ZYCh`k(`lD<{JbEYz+X51PM#N9&Cw#}UtAn5VZWu(HK1>0_$%*^ zsO4g(+%#X%+69b@swvXHYA?XtXzYahB7B1j{1yI+?5Xv_jX{W3`YU;#u*i@OMtNL4 zv2^b0RexN)^5^znQ?L6@sJ)ERvTzAJeNIiA{>4*_kd?<~D}Un8g=zJ$(%Wm%+iAL; zSosZ6c>h|pH<-cIhBmD8ul29fyDx&CuIu^q>aW#)jaK=K{8c@_vO2{sni?gBZpX4atMgkQ?Roh1GefqlfDv{_>tLu1@2x zQX$4l%F=4-Bs26DF}hXuv^={0BOs6!deU*qM;Xbmq^0?R@+<`p^&4sp`%WO2SJk`p zujG?@RPB}dzl+goc)N=k&@J_^@mFU&Sp!q2gX2-UT*b@%iL@AA?q60*f0l8SZLF%} zB3kpT5{loc_ePJd_pjIb60HFLsPV6FrU??|AX*%)_19{tKq0~Lvd|5GjXr08?W$br zbae2PzlPCIDO*nL&1rwF)C&rG#$R`Sw>~e)kEeZUI*L(IqAyv+I#5jqwE7n5tSU`a zfp&F)`!PLc1#i*S?Rhl^^iW!JeBo@7bX-5$sg{qC#9#CmNhh9^7Vo?D?=|RxUuEb& z{VP*4*qqb5&FJcQ9;r|2^Icg-9fmS1g6`XavxqN{E}UHhxapz{bSVO9n4c|HVe~lO zBwf^{F3FzSU#zto!fQsm2h(3azJI0Q`QXa8`en=f#Tj@L0^STghA$8Q^Ty8#%%l=7 z>HFqvxzo5wQF=t*j)9lm;^K9QFGhhg&v|0&Z(4D>%*Zx|bJBP>L5D9c!(TX~kNxtnEN zi9%D_l7(4=N+FCW__dIG=wFBN_{Y0UYX!J8jUzzAiNzKMv6~R>qLp&fmP*)W3h@WD z(n>64SJ!?^v|@{AEOI(V+cHPSe8Gz}C&;3au)bF#ZR3ocD`=lmg|UraIh)Ifw(*#O0;SlSh9slcd=b3X-g`23z4T8Ii0a}=z{WfxM+t-h>@4ahKrc6yyF) zN>UQ{E^+-D8T`$S|cOYTL5u=Fa+o!7|G z(t=~ufEnl^;_+beO=htz6tQ*0-Da|sT0vAD>i zp|mW-Mm2T=RX|4is3k9I!#?HTAiPREcu9_ohR6A~SmDdqvT-P;8%iilX zrEG(jBH=+mg=RTF?XxuT$7RtLNJk+A>vza(F5(->}mjQDaU<-(1E@+Bjg_!PfPi0Dj=1$ktCnI;*gKlo)D*>4IF zP{TBwxvvxV(!{DSY}|tu3d+in+)uM~6j09Mo%uFW#-N>X?@Y(xw?8u*N)~zHQTDr^ zQT#IV3k|)Usp5kQ#6X)4X>d$@K>-x&K9z)Gi(ah6O+#2)7%}~c$WV;AOakW+_=$c8 z=$W%E#;?*Sb$`TNaK_|Kf`^9UCrd*prcqLiyF|OPWEfRIT{J9P(YocgH+3hDeDb=J z+vw^_u3Cx)6IF&sBA`0)nCW|LscmFyX`1lTeIRv7w<0(`)zFnSWr#DdEw^l*kV} z(MF&0q=G0HZQ#j?8If}MF7Z*5@F=54`JB0!Q`oCVVMOWY6x6D+mISRIuds@QXO#9a z)g38YpL7)4I2}%`Rpb^CtJKHHD<1+76PJiolkngSV=*AO*M}1!nU-@8)B1KXM!*s+ zjq1)`Ae0RhKHVUml(g->#N$W1I{5R}x(2TjZj63Ol;Jy-fP);-*POdGz^A^QeCkyy zLX}R~RX&wiPa+S%Gm1sFXtQkMnxhb$!ict2whizm3V2JE6mSv~GhOZpq1$PIL%m6p zZ=mIs{A>e9A?DD<88HWH5;0dd*ZLgtP(ADk6<2xK%TcuW5z0(c>E=^^yyebjY>@qK z%icfTqEP8I0^@@Su1s#;70;JoX!P z43<%E>IofB_9i@xAfqzsk?>>&UZXQvKsI}-24Ev`!lU#Sk1y6g;UWuv!WyLJjjxPY zfzm`RHrAo=I2K?_zr<@Z!68u(d7k|G2|tzn=_!(cw(Vi@0DO6X#zX^T0^+@JbR2_Z zie2C_6c8<-#Vs>Y(YteJ_b-UwcFQhwEMG^!Lr865i?=0m&=s^O0&Oenkf0v;_a8iS z<^&BYEV;lJWB}NPWQ=#n)x#kO)uOT&rdSsf8+iI%4T)`zZiTHX&VjUCC#ez~_ zRExi$6XP}8DWM%BC`Ri^r1B3O?cOgvKhSZux1;+sHo-g1g2U0vQnr)j$V zEjqnPr#I;IQ#$>UPG7?boe&EYC8Fm_gWo-J>&W+BSZIE3xoXGl^*?I-e&bJeF6``1?l_q|IhtHMhIPU%YbXLb zYIt8O*Ol}gxZCx>Xjm6DC<%;~D+YviTb#UV$u%32n>&*|XO`E-wJfgcvT%N9{r6(G zH~mol(JMcA?G?-JisjtX4{QtuW{VXIzq$&R zoR#y=$_3}Ti9@t|@{t=ymR#PX%Zud^uo=u^nRlY&{kE;Y-}H9VQrp4#wu5&LF18(6 zD0yMN?Z|BLPrLrO>!&Ba`st~0S`XkiKaGd;B`-`I`ohWOn*Xn?^N(rcJmdHs9N0b^ zW9+ky!NwQ}m|unv0tpZj2Z)JV643%_Lx{((@Vf((9}}U2wUz4rkYdtxm?cw=wNfrx zq`1?hFfG!ER4G)N)Ysfy&LmW@HPW>IAc@6NY0{qeaM`9w*7xq-_q}`1=kt5_Ja^CM z$-iCp+cI9;vRD=}eq9tf`1-X=flG^J_~q#q$OtXR9yvMYp5=}uT+J8OB4^#w>nqH+ zmTxS=n_c|L^QhT}v_BCBhNXdt$iRfC^slAro~5PF+ppVM=C*Co#;29ddiU~0AoG5K zs(>%0H_m!D6!iO(5!w^Jcz@>iGd%72aPq^+_x`6uQg~Jsp7qqskQX_RN~yq+&!3$;8&gmaC`1#CF>1+`EX5H^F)HciEj^NDBw`uiEn@*~RBc&RJGiVR zQYx$w3TrT|Kz~l?y&(0DMS90X%{yxbCTh%O-J&r+K<`jslE`++OIl`TSm982Ep&xW zBf3POO4lhB(pc|L-E4=ztf?=K3bq5tgWt_Y^aX($*sTtA-0EEDgrjG_W9My$k+lZV zwE|Taua$qh&?&As+;@NUtI_cJ<(iegZ%4ivS?NM_r$C+CZFg_q!rSOz8?v?|x;R)1~i);AZl7B`_h1^G=|&yBlB!Qs z^`XI0{ObC}`I)&H09~wE?CXwV&WjEW2#&!QiHQ#Fnqx4;#N)IiLqWt)z#9rBLq)_; zAwU{&L(>lAwKN8)h+}XYw|_w-W~2tZfKxcsNG^L($60i)o3DN+qNxnDaG3z|5H&eC zcTifeQ`BS!ngHPdf7>+k%~FnDu6%Ul;SoNgRn)Zxli^4C)U&BJhL~IS1-qbjt`{F( zDq8MaX+k}NeDP33;|#b1)1l7)-)MYQ1~?z57fuW6;*DHqz`F~Y7BqHlSaP`|E;sLL z_@@kBwnj8oL6fzvNxkO%2-@u}>w{H-#<8O$EcUtmtb3=Hp!>;HRo`|!?D8+Cfc4-y z$(y|p?-xi~%PZvd#}=DHjZ*1RvGnMCWw3IwDWE|5W6yq!+Pm~c4cE=O!paARr89rB zE@{z;A*3A^lq2!LzJqrS5q!tYm$dSQt$gkqDCbS2Z4;F3(BMTg5fr^WO1_xmAd%4m80Z$CmR#m1Nd#D`76!2xAY#6{_dd>yU;%?=Bs#;hl}k zeUC;Tj-s}=q&8oq&4;@CrS40S?n^)fNz-o{%+He&($cqq3zD;?B2pNj&6>3@wM1r) zl;K*t{thh3$11>!ytPq3c7Rjj$fkR2=YZ>?;XYh6;XFA9e&9H>zrG8ba+i zGT1)RQpEK=POn%d11I4uj_NZ)l`I)@vW_nsxTG(<+yq-6PG8Ccx@;x@M4Cu*)SS)g zKc8kBk*Scg{-Ga6ei z&6+s1=%|D%Uv@RBej91e3d+vCwLQP_ajnQ&hUjvE+P@LUnPjtBD7ze$H!gQApN^mD z@^E^yovq_-<$JyDZU;(hd0QE>mLnQpJ2n)C-M+sxveV%avB1qcj-%{`<%;Fr6+oyA zh%E!+i9s}SMLcl@{p>1gnnu$0(JcF3ccJ%^@GHv*Z4!tsQM&2IgYdwg3=!~IoMpul#fjLffMk90YUn*WG;x9 z3y{glC2&tMJYS__)G zO?_5qQqt!loU{z~AgBuEOBqFxj3Q*gl7WR5-1A*Xa1XigAalB6l!gkTF*yX&x_jgqSCjCQ>kSFn@3RMo{+W~lBsRtJazGn zZRkw5N&@>4Rm!U zhso=3Bm1b-JYGeus2HpVG~>0@H&im{YE8~WY7%cfI6-IxQ!-ZT@lq&~Xm=h=qK8eO zYpI5bB;{H`GS<@LB~Y|(ajPe2`nx=L4J>WgapO(Ood}b^YLc*;4uk$p=f^^gx+%&vFmwTQ93Bo(E?})qC4u(i z$(8<|aVQ(={V8bA7@o(tJ<#z=td)3vyzc^T3KqdsvOKAP!iyL2CVvVFRo#UWImrCyMFvL*oMf3xhHxsCjmSLj_Hf8IKE9b40lb~1b@H(6WkWN zJN>dXYQuBapbQ>Hf~#77@(5-Qejwn-zDzk*-w)ncZ18~d91s)!zu@xs@pi6b$-#@V zjfvk|@w>5jv7ExvhQ)_v5X&%@c`UcE+{GfBhO&Js+lR8<7sC}lVEGponIa&Ym=4?} zrku&K{JT~vHfsF%wWcQ&Fc~k(j~{j;GU1&wuv+WDU3h7x zv170k=Skp&C#ZK4(#fM+1jvg-Slp5qtKN&$mG4Fg*?BpQ4-Px#u!L-7SqZ50GW$+nn|2gA*Kgjd`u^Ez+*iL t1!Q)tjfAN4m Date: Mon, 24 Aug 2026 14:28:48 -0700 Subject: [PATCH 07/16] ADFA-5153: Fix eight review findings, one of which made the output unreadable jatezzz, high: the script recompressed content against the dictionary and never declared a version, so the app -- which gates on the declared MAJOR, not on the presence of CompressionDictionary -- would refuse to attach the dictionary and every row just migrated would fail to decode. A database that looks migrated and serves nothing. The migrate phase now writes major 2 in the same transaction as the last batch, and says so; a dry run says what it would declare and why. Declared even after a partly failed run, since WebServer falls back to a plain decode for rows that did not migrate but cannot read the ones that did without it. jatezzz, medium: newly inserted continuation rows carried the base row's languageID, while WebServer looks continuations up with "languageId = 1" hardcoded. Any item whose base row is not language 1 would have had its continuations become invisible and its page truncate at 1 MiB -- the exact ADFA-5171 symptom this script exists to remove. Continuations are inserted as language 1 now. jatezzz, medium: retyping into a type whose own compression is not 'none' left the bytes compressed under a type they did not match, produced two spurious verifier errors per row, and could have served raw compressed bytes. Such a target is now refused with an actionable message and the row left alone. jatezzz, low: one missing or NULL-content row aborted the whole run from inside a worker, with earlier batches committed and no summary. Both phases now report it and continue. Testing that found a second path to the same crash: NULL content also made LENGTH() NULL, so the phase summary threw before any row was read. jatezzz, low: phase 1 submitted every candidate at once and held every decoded plaintext resident. It batches now, like phase 3. CodeRabbit, major: renumber_item parked rows under "{base}-renumbering-{n}", which a real row can already occupy -- failing on UNIQUE(path) after the collision checks had passed. A single ascending pass needs no temporary names: contiguity is verified first, so the lowest target is free and every later one was vacated by the move before it. My comment claiming otherwise was wrong. CodeRabbit, major: verify_retype reported valid -2 suffixes as errors in a retype-only run, so a successful run exited 1. That check now runs only when renumbering was requested. Co-Authored-By: Claude Opus 5 --- .../migrate_content_to_dictionary_brotli.py | 228 +++++++++++++----- 1 file changed, 171 insertions(+), 57 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index d46b80a9c0..9ff51ef870 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -84,6 +84,9 @@ CHUNK_BYTES = 1024 * 1024 +# WebServer looks continuations up with "languageId = 1" hardcoded, whatever the base row says. +CONTINUATION_LANGUAGE_ID = 1 + def is_text_type(value: str) -> bool: """Whether a ContentTypes.value is a text type, matched at the boundary. @@ -322,7 +325,9 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: rows = connection.execute( f""" SELECT C.id, C.path, C.languageID, C.contentTypeID, C.templateId, - LENGTH(C.content), CT.value, CT.compression + -- IFNULL: a row with NULL content has NULL length, which made the byte totals + -- (and so the phase summary) throw before read_blobs could report the row. + IFNULL(LENGTH(C.content), 0), CT.value, CT.compression FROM Content C JOIN ContentTypes CT ON CT.id = C.contentTypeID WHERE {predicate} @@ -362,11 +367,17 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: return sorted(items.values(), key=lambda item: item.base_path) -def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes]: +def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes] | None: + """An item's slices in order, or None when a row has vanished or holds NULL content. + + None rather than an exception: a single unreadable row used to abort the whole run from + inside a worker, leaving earlier batches committed and printing no summary at all. + """ ids = [item.base_id] + [row_id for row_id, _, _ in item.continuations] placeholders = ",".join("?" * len(ids)) found = dict(connection.execute(f"SELECT id, content FROM Content WHERE id IN ({placeholders})", ids).fetchall()) - return [found[row_id] for row_id in ids] + blobs = [found.get(row_id) for row_id in ids] + return None if any(blob is None for blob in blobs) else blobs def write_item( @@ -406,7 +417,12 @@ def write_item( INSERT INTO Content (path, languageID, content, contentTypeID, templateId) VALUES (?, ?, ?, ?, ?) """, - (f"{item.base_path}-{suffix}", item.language_id, payload, type_id, item.template_id), + # languageID 1, not the base row's: WebServer's continuation query is + # "WHERE path = ? AND languageId = 1" (documented in + # docs/documentation-database.md), so a continuation inserted under any other + # language is invisible and the page truncates at its first 1 MiB -- the exact + # ADFA-5171 symptom this script exists to remove. + (f"{item.base_path}-{suffix}", CONTINUATION_LANGUAGE_ID, payload, type_id, item.template_id), ) inserted += 1 else: @@ -433,6 +449,58 @@ def retype_rows(connection: sqlite3.Connection, item: Item, content_type_id: int ) +# The MAJOR the app requires before it will attach the dictionary at all +# (DatabaseVersionResolver.MAJOR_VERSION_WITH_COMPRESSION_DICTIONARY). Migrating content without +# declaring this leaves a database whose every brotli row fails to decode: WebServer gates on the +# declared version, not on the presence of CompressionDictionary, so it never attaches the +# dictionary and the plain decode then throws "corrupted input" on every migrated row. +DICTIONARY_MAJOR_VERSION = 2 + +VERSION_TABLE_SQL = """ +CREATE TABLE IF NOT EXISTS DocumentationDatabaseVersion ( + major INT NOT NULL, + minor INT NOT NULL, + patch INT NOT NULL, + who TEXT NOT NULL, + comment TEXT NOT NULL, + changeTime TIMESTAMP DEFAULT CURRENT_TIMESTAMP +) +""" + + +def declared_major(connection: sqlite3.Connection) -> int | None: + """The MAJOR this database declares, or None when it declares none. + + Reads the row with the highest rowid: the table holds exactly one row by contract, and this is + the row both the app's DatabaseVersionResolver and docdb-studio read (ADFA-5220). + """ + if not table_exists(connection, "DocumentationDatabaseVersion"): + return None + row = connection.execute( + "SELECT major FROM DocumentationDatabaseVersion ORDER BY rowid DESC LIMIT 1" + ).fetchone() + return row[0] if row is not None and row[0] is not None else None + + +def declare_dictionary_version(connection: sqlite3.Connection) -> None: + """Records that this database's brotli content is dictionary-compressed. + + Written in the same transaction as the last batch of content, because the two facts have to + travel together: content compressed against the dictionary, and a version saying so. Exactly + one row, replaced rather than appended, matching populate_db.py. + """ + connection.execute(VERSION_TABLE_SQL) + connection.execute("DELETE FROM DocumentationDatabaseVersion") + connection.execute( + "INSERT INTO DocumentationDatabaseVersion (major, minor, patch, who, comment) VALUES (?, 0, 0, ?, ?)", + ( + DICTIONARY_MAJOR_VERSION, + "migrate_content_to_dictionary_brotli.py", + "Content rows compressed against CompressionDictionary", + ), + ) + + def table_exists(connection: sqlite3.Connection, name: str) -> bool: """Whether `name` is a table in this database -- checked the way WebServer does.""" found = connection.execute( @@ -468,15 +536,13 @@ def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> st return f"{target} already exists and belongs to another row; left alone" if write: - # Two passes: park every row under a name nothing can collide with, then - # settle them into their new slots. One ascending pass would be enough only - # if the target were always already free, and for a shift of 1 it never is. - for row_id, suffix, _ in item.continuations: - connection.execute( - "UPDATE Content SET path = ? WHERE id = ?", - (f"{item.base_path}-renumbering-{suffix}", row_id), - ) - for row_id, suffix, _ in item.continuations: + # One ascending pass, no temporary names. The suffixes were just verified + # contiguous, so the lowest target (first_suffix - shift) is free -- nothing + # occupies a suffix below first_suffix -- and every later target was vacated by + # the move before it. The parking pass this replaces invented + # "{base}-renumbering-{n}" paths that a real row could already hold, which would + # fail on UNIQUE(path) after the collision checks above had passed. + for row_id, suffix, _ in sorted(item.continuations, key=lambda entry: entry[1]): connection.execute( "UPDATE Content SET path = ? WHERE id = ?", (f"{item.base_path}-{suffix - shift}", row_id), @@ -518,30 +584,40 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s retyped: list[tuple[Item, Inspection, str]] = [] before_total = after_total = 0 - pending = {pool.submit(inspect_item, item, read_blobs(connection, item)): item for item in items} - for future in futures.as_completed(pending): - item = pending[future] - found = future.result() - if found.status == "error": - errors.append(f"{item.base_path}: {found.detail}") - continue - if found.status == "keep": - counts["left as text"] = counts.get("left as text", 0) + 1 - notes.append(f"{item.base_path}: {found.detail}") - continue - - target = found.sniffed - if target == "video/quicktime" and args.mov_type == "mp4": - target = "video/mp4" - extension = item.base_path.rsplit(".", 1)[-1].lower() - if extension not in target and not (extension in ("jpg", "jpeg") and target == "image/jpeg") \ - and not (extension == "mov" and target.startswith("video/")): - notes.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") - - retyped.append((item, found, target)) - counts[target] = counts.get(target, 0) + 1 - before_total += found.before - after_total += found.after + # Batched like phase 3, and for the same reason: this holds every candidate's decoded + # plaintext resident until the write loop, and one row in this database is 23 MB. + for offset in range(0, len(items), args.batch): + pending = {} + for item in items[offset : offset + args.batch]: + blobs = read_blobs(connection, item) + if blobs is None: + errors.append(f"{item.base_path}: a row is missing or holds NULL content; left alone") + continue + pending[pool.submit(inspect_item, item, blobs)] = item + + for future in futures.as_completed(pending): + item = pending[future] + found = future.result() + if found.status == "error": + errors.append(f"{item.base_path}: {found.detail}") + continue + if found.status == "keep": + counts["left as text"] = counts.get("left as text", 0) + 1 + notes.append(f"{item.base_path}: {found.detail}") + continue + + target = found.sniffed + if target == "video/quicktime" and args.mov_type == "mp4": + target = "video/mp4" + extension = item.base_path.rsplit(".", 1)[-1].lower() + if extension not in target and not (extension in ("jpg", "jpeg") and target == "image/jpeg") \ + and not (extension == "mov" and target.startswith("video/")): + notes.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") + + retyped.append((item, found, target)) + counts[target] = counts.get(target, 0) + 1 + before_total += found.before + after_total += found.after missing = sorted({target for _, _, target in retyped if target not in types}) for value in missing: @@ -556,20 +632,28 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s print(f" ContentTypes would insert {value} (compression none)") inserted_total = deleted_total = 0 + kept = [] for item, found, target in retyped: type_id, compression = types[target] + # A target type whose own compression is not 'none' cannot receive plaintext. Retyping + # into it used to leave the bytes compressed and the row declaring a type they are not, + # which the verifier below then reported twice -- and if WebServer does not handle that + # compression at all, the row would serve raw compressed bytes to a browser. + if compression != "none": + errors.append( + f"{item.base_path}: {target} is registered with compression '{compression}', not 'none'; " + f"left as {item.content_type}. Fix the ContentTypes row, then re-run" + ) + counts[target] = counts.get(target, 0) - 1 + continue + kept.append((item, found, target)) if not write: continue - if compression == "none": - inserted, deleted = write_item( - connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id - ) - inserted_total += inserted - deleted_total += deleted - else: - # The honest type is itself a compressed one, so the stored bytes stay - # as they are and phase 3 picks the row up. - retype_rows(connection, item, type_id) + inserted, deleted = write_item( + connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id + ) + inserted_total += inserted + deleted_total += deleted if write: connection.commit() @@ -579,7 +663,7 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s f"({'+' if after_total >= before_total else ''}{human(after_total - before_total)})") if write: print(f" rows inserted {inserted_total} deleted {deleted_total}") - return {item.base_path for item, _, _ in retyped}, errors, notes + return {item.base_path for item, _, _ in kept}, errors, notes def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> tuple[int, list[str], list[str]]: @@ -624,7 +708,7 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> return fixed, errors, notes -def verify_retype(connection, retyped_paths: set[str], mov_type: str) -> list[str]: +def verify_retype(connection, retyped_paths: set[str], mov_type: str, expect_renumbered: bool) -> list[str]: """Re-read what phase 1 wrote and confirm the bytes match the declared type.""" problems: list[str] = [] written = {item.base_path: item for item in load_items(connection, "1 = 1")} @@ -642,7 +726,9 @@ def verify_retype(connection, retyped_paths: set[str], mov_type: str) -> list[st problems.append(f"{path}: declared {expected} but the stored bytes sniff as {found or 'unknown'}") if item.compression != "none": problems.append(f"{path}: retyped to {expected}, whose compression is {item.compression}") - if item.continuations and item.suffixes != list(range(1, len(item.continuations) + 1)): + # Only when this run was asked to renumber: a retype-only run legitimately leaves + # -2-based numbering alone, and reporting it as an error made a successful run exit 1. + if expect_renumbered and item.continuations and item.suffixes != list(range(1, len(item.continuations) + 1)): problems.append(f"{path}: continuations numbered {item.suffixes}, expected 1..n") return problems @@ -729,7 +815,9 @@ def select(items: list[Item]) -> list[Item]: if write: # A verification failure means the bytes and their declared type disagree # after we wrote them -- the most serious thing this script can report. - errors += verify_retype(connection, retyped_paths, args.mov_type) + errors += verify_retype( + connection, retyped_paths, args.mov_type, expect_renumbered="renumber" in args.phase_list + ) print() if "renumber" in args.phase_list: @@ -765,12 +853,22 @@ def select(items: list[Item]) -> list[Item]: for offset in range(0, len(items), args.batch): batch = items[offset : offset + args.batch] - pending = { - pool.submit( - migrate_item, item, read_blobs(connection, item), args.quality, args.window, args.only_if_smaller - ): item - for item in batch - } + pending = {} + for item in batch: + blobs = read_blobs(connection, item) + if blobs is None: + # Reported, not raised: this used to surface as a TypeError inside a worker + # and abort the run with earlier batches already committed and no summary. + counts["error"] += 1 + failed_items.append( + Result(item.base_path, "error", detail="a row is missing or holds NULL content") + ) + continue + pending[ + pool.submit( + migrate_item, item, blobs, args.quality, args.window, args.only_if_smaller + ) + ] = item for future in futures.as_completed(pending): item = pending[future] @@ -831,10 +929,26 @@ def select(items: list[Item]) -> list[Item]: report(errors, notes) if write: + # The version goes in with the content, not after it: a database holding + # dictionary-compressed rows while declaring anything below + # DICTIONARY_MAJOR_VERSION is one the app refuses to attach the dictionary for, so every + # row just migrated fails to decode. Declared even on a partly failed run -- WebServer + # tries the dictionary first and falls back to a plain decode, so a row that did not + # migrate still serves, while the ones that did only serve with this row present. + before = declared_major(connection) + if before is None or before < DICTIONARY_MAJOR_VERSION: + declare_dictionary_version(connection) + print(f"declared database version {DICTIONARY_MAJOR_VERSION}.0.0 " + f"(was {'none' if before is None else before})") connection.commit() print("\nRun VACUUM to reclaim the freed pages: sqlite3 %s 'VACUUM;'" % args.database) else: connection.rollback() + before = declared_major(connection) + if before is None or before < DICTIONARY_MAJOR_VERSION: + print(f"would declare database version {DICTIONARY_MAJOR_VERSION}.0.0 " + f"(currently {'none' if before is None else before}) -- without it the app will not " + f"attach the dictionary and every migrated row fails to decode") print("\nNothing written. Re-run with --yes on a copy to apply.") connection.close() From 0be5a445da3cd651404e59c1c9dbe8077da3e0d4 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Mon, 24 Aug 2026 15:22:47 -0700 Subject: [PATCH 08/16] ADFA-5153: Finish three fixes the review found half-done All three follow from the previous round and are the reviewer's, not mine. Batching phase 1's submissions bounded the workers, not the memory: every completed Inspection was appended to a list and written only after the last batch, so --batch did nothing about the thing that actually runs out. Each batch is now inspected, typed, written and committed before the next one starts, and each item's decoded payload is dropped as soon as it is written. The verifier gets a set of paths rather than a list holding slices. The continuation language fix only covered inserts. Reusing an existing continuation row updated its content and type but left its languageID, so a row that predates this script and carries the base row's language stayed invisible to WebServer's continuation query -- the same truncation, through the row this script chose not to replace. The update normalises it too. The version was declared after the last batch, so a run interrupted between two committed batches left dictionary-compressed rows in a database still declaring a version the app will not attach the dictionary for: every committed row would fail to decode. It is now declared in the same transaction as the first batch of migrated content, with the end-of-run declaration kept as the fallback for a run that migrates nothing but finds content already migrated. Verified: a 4-row database with a realistic dictionary declares 2.0.0 during the batch loop rather than after it; retype with --batch 1 still retypes and still refuses a compressed target type; and the earlier round's checks all still hold -- the parking-path squatter, the NULL-content row, language 1 on inserted continuations, no suffix complaints in a retype-only run, and a dry run that leaves the file byte-identical. Co-Authored-By: Claude Opus 5 --- .../migrate_content_to_dictionary_brotli.py | 109 +++++++++++------- 1 file changed, 68 insertions(+), 41 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 9ff51ef870..825d44c10c 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -427,8 +427,12 @@ def write_item( inserted += 1 else: connection.execute( - "UPDATE Content SET content = ?, contentTypeID = ? WHERE id = ?", - (payload, type_id, row_id), + # languageID too, not just the bytes: a continuation row that predates this script + # can carry the base row's language, and WebServer's continuation query filters on + # languageId = 1 -- so reusing the row without normalising it leaves the page + # truncated exactly as an unnumbered continuation would. + "UPDATE Content SET content = ?, contentTypeID = ?, languageID = ? WHERE id = ?", + (payload, type_id, CONTINUATION_LANGUAGE_ID, row_id), ) # Whatever is left over described slices the new stream no longer needs. @@ -586,6 +590,12 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s # Batched like phase 3, and for the same reason: this holds every candidate's decoded # plaintext resident until the write loop, and one row in this database is 23 MB. + inserted_total = deleted_total = 0 + kept: set[str] = set() + + # Each batch is inspected *and written* before the next one starts. Batching only the + # submissions still held every decoded plaintext until a write loop at the end, so --batch + # bounded the workers and not the memory, which is the thing that runs out. for offset in range(0, len(items), args.batch): pending = {} for item in items[offset : offset + args.batch]: @@ -614,48 +624,46 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s and not (extension == "mov" and target.startswith("video/")): notes.append(f"{item.base_path}: named .{extension} but the payload is {found.sniffed}") - retyped.append((item, found, target)) + if target not in types: + if write: + cursor = connection.execute( + "INSERT INTO ContentTypes (value, compression) VALUES (?, 'none')", (value := target,) + ) + types[value] = (cursor.lastrowid, "none") + print(f" ContentTypes + id {cursor.lastrowid} {value} (compression none)") + else: + types[target] = (-1, "none") + print(f" ContentTypes would insert {target} (compression none)") + + type_id, compression = types[target] + # A target type whose own compression is not 'none' cannot receive plaintext. Retyping + # into it would leave the bytes compressed under a type they are not, which the verifier + # then reports twice -- and if WebServer does not handle that compression at all, the row + # serves raw compressed bytes to a browser. + if compression != "none": + errors.append( + f"{item.base_path}: {target} is registered with compression '{compression}', not 'none'; " + f"left as {item.content_type}. Fix the ContentTypes row, then re-run" + ) + continue + counts[target] = counts.get(target, 0) + 1 before_total += found.before after_total += found.after + kept.add(item.base_path) - missing = sorted({target for _, _, target in retyped if target not in types}) - for value in missing: - if write: - cursor = connection.execute( - "INSERT INTO ContentTypes (value, compression) VALUES (?, 'none')", (value,) - ) - types[value] = (cursor.lastrowid, "none") - print(f" ContentTypes + id {cursor.lastrowid} {value} (compression none)") - else: - types[value] = (-1, "none") - print(f" ContentTypes would insert {value} (compression none)") + if write: + inserted, deleted = write_item( + connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id + ) + inserted_total += inserted + deleted_total += deleted + # found.slices is the only large thing here; dropping the reference lets this batch's + # plaintext be collected before the next batch decodes its own. + found.slices = [] - inserted_total = deleted_total = 0 - kept = [] - for item, found, target in retyped: - type_id, compression = types[target] - # A target type whose own compression is not 'none' cannot receive plaintext. Retyping - # into it used to leave the bytes compressed and the row declaring a type they are not, - # which the verifier below then reported twice -- and if WebServer does not handle that - # compression at all, the row would serve raw compressed bytes to a browser. - if compression != "none": - errors.append( - f"{item.base_path}: {target} is registered with compression '{compression}', not 'none'; " - f"left as {item.content_type}. Fix the ContentTypes row, then re-run" - ) - counts[target] = counts.get(target, 0) - 1 - continue - kept.append((item, found, target)) - if not write: - continue - inserted, deleted = write_item( - connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id - ) - inserted_total += inserted - deleted_total += deleted - if write: - connection.commit() + if write: + connection.commit() for target, count in sorted(counts.items(), key=lambda kv: -kv[1]): print(f" {target:22} {count:>4}") @@ -663,7 +671,7 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s f"({'+' if after_total >= before_total else ''}{human(after_total - before_total)})") if write: print(f" rows inserted {inserted_total} deleted {deleted_total}") - return {item.base_path for item, _, _ in kept}, errors, notes + return kept, errors, notes def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> tuple[int, list[str], list[str]]: @@ -844,6 +852,8 @@ def select(items: list[Item]) -> list[Item]: print() counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} + wrote_migrated_content = False + version_declared = False before_total = after_total = 0 inserted_total = deleted_total = 0 # Not named `errors`: that name already holds this run's phase 1 and 2 failures, @@ -885,8 +895,23 @@ def select(items: list[Item]) -> list[Item]: inserted, deleted = write_item(connection, item, result.slices, renumber=False) inserted_total += inserted deleted_total += deleted + wrote_migrated_content = True if write: + # In the same transaction as the first batch of migrated content, not after the last + # one. A run interrupted between two committed batches would otherwise leave the + # database holding dictionary-compressed rows while declaring a version below the one + # the app requires -- so it would decline the dictionary and every committed row + # would fail to decode. Declaring first means the worst an interruption leaves is a + # partly migrated database that still serves, since the app falls back to a plain + # decode per row. + if wrote_migrated_content and not version_declared: + before = declared_major(connection) + if before is None or before < DICTIONARY_MAJOR_VERSION: + declare_dictionary_version(connection) + print(f"\ndeclared database version {DICTIONARY_MAJOR_VERSION}.0.0 " + f"(was {'none' if before is None else before})") + version_declared = True connection.commit() done = min(offset + args.batch, len(items)) @@ -935,8 +960,10 @@ def select(items: list[Item]) -> list[Item]: # row just migrated fails to decode. Declared even on a partly failed run -- WebServer # tries the dictionary first and falls back to a plain decode, so a row that did not # migrate still serves, while the ones that did only serve with this row present. + # Anything already dictionary-compressed still needs the declaration, even when this run + # migrated nothing itself (every row came back "already"). before = declared_major(connection) - if before is None or before < DICTIONARY_MAJOR_VERSION: + if not version_declared and (before is None or before < DICTIONARY_MAJOR_VERSION): declare_dictionary_version(connection) print(f"declared database version {DICTIONARY_MAJOR_VERSION}.0.0 " f"(was {'none' if before is None else before})") From 45a8aa6068b775c9c519350650317078ce102852 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Tue, 25 Aug 2026 18:47:28 -0700 Subject: [PATCH 09/16] ADFA-5153: Only renumber items that are actually chunked, and refuse a taken path Two ways this script could damage content, both found in review. phase_renumber decided an item was chunked from the path suffix alone. A real page whose greedy base happens to be another real page -- k/kotlin-1-2 under k/kotlin-1 -- was therefore renamed to k/kotlin-1-1, which 404s every link to it and leaves the base looking like a two-slice item. The signal that would have prevented it, base row length == CHUNK_BYTES, was already computed twenty lines below as a note, i.e. after every rename had been made, and report() caps notes at 30. That test now selects the candidates instead: an item is chunked when its base row is exactly CHUNK_BYTES, which is what the app's own continuation query requires before it will reassemble anything. A numeric-suffixed sibling with a differently sized base is reported as independent content and left alone. write_item INSERTed continuation paths with no UNIQUE(path) check, though renumber_item already makes exactly that check before it moves anything. An occupied target -- a foreign row, or a continuation the phase predicate excludes -- surfaced as a bare sqlite3.IntegrityError from the middle of a phase, with earlier batches committed and no summary printed, which is the failure the batching was introduced to avoid. write_item raises PathClash before it writes anything now, and both call sites record it as an error for that item and carry on with the rest. Verified against synthetic databases holding each case: - k/kotlin-1 (4 bytes) + k/kotlin-1-2: left alone, reported as independent. Against the script as it stood, k/kotlin-1-2 is renamed to k/kotlin-1-1. - big/page (exactly CHUNK_BYTES) + -2 + -3: still renumbered to start at -1, so the repair this phase exists for is unaffected. - img.gif with continuations at -2/-3 while another content type owns img.gif-1: PathClash, named, instead of an IntegrityError mid-run. Found in review of PR #1724. --- .../migrate_content_to_dictionary_brotli.py | 73 +++++++++++++++---- 1 file changed, 59 insertions(+), 14 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 825d44c10c..f0c690413b 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -380,6 +380,22 @@ def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes] | None return None if any(blob is None for blob in blobs) else blobs +class PathClash(Exception): + """A continuation path this item needs is owned by some other row.""" + + +def continuation_clash(connection: sqlite3.Connection, item: Item, slices: int, renumber: bool) -> str: + """The first target path owned by a foreign row, described; '' if the write is safe.""" + start = 1 if renumber else item.first_suffix + own = {row_id for row_id, _, _ in item.continuations} + for suffix in range(start, start + slices - 1): + target = f"{item.base_path}-{suffix}" + row = connection.execute("SELECT id FROM Content WHERE path = ?", (target,)).fetchone() + if row and row[0] not in own: + return f"{target} already exists and belongs to another row" + return "" + + def write_item( connection: sqlite3.Connection, item: Item, @@ -387,7 +403,17 @@ def write_item( renumber: bool, content_type_id: int | None = None, ) -> tuple[int, int]: - """Write an item's new slices back. Returns (rows inserted, rows deleted).""" + """Write an item's new slices back. Returns (rows inserted, rows deleted). + + Raises PathClash if a continuation path is owned by a foreign row. renumber_item makes the + same check before it moves anything; this one did not, so an occupied path surfaced as a bare + sqlite3.IntegrityError out of the middle of a phase, with earlier batches already committed + and no summary printed -- the failure mode the batching was introduced to avoid. + """ + clash = continuation_clash(connection, item, len(slices), renumber) + if clash: + raise PathClash(f"{item.base_path}: {clash}; left alone") + type_id = item.content_type_id if content_type_id is None else content_type_id connection.execute( "UPDATE Content SET content = ?, contentTypeID = ? WHERE id = ?", @@ -653,9 +679,14 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s kept.add(item.base_path) if write: - inserted, deleted = write_item( - connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id - ) + try: + inserted, deleted = write_item( + connection, item, found.slices, renumber="renumber" in args.phase_list, content_type_id=type_id + ) + except PathClash as clash: + errors.append(str(clash)) + found.slices = [] + continue inserted_total += inserted deleted_total += deleted # found.slices is the only large thing here; dropping the reference lets this batch's @@ -684,14 +715,22 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> items = select([item for item in load_items(connection, "1 = 1") if item.continuations]) if args.renumber_scope == "retyped": items = [item for item in items if item.base_path in retyped_paths] - broken = [item for item in items if item.first_suffix != 1] + # A "-" sibling is a naming coincidence until the base row proves otherwise. The app + # decides an item is chunked by its base row being exactly CHUNK_BYTES (WebServer's continuation + # query), so that is the test here too. Without it this phase renamed real, independent pages: + # a page k/kotlin-1-2 whose greedy base is the real page k/kotlin-1 was renumbered to + # k/kotlin-1-1, which 404s every link to it and leaves the base looking like a 2-slice item. + # The check used to happen 20 lines below, as a note, after every rename had been made. + chunked = [item for item in items if item.base_bytes == CHUNK_BYTES] + coincidental = [item for item in items if item.base_bytes != CHUNK_BYTES] + broken = [item for item in chunked if item.first_suffix != 1] starts = sorted({item.first_suffix for item in broken}) if broken: - print(f"[2/3] renumber {len(broken)} of {len(items)} chunked items start at " + print(f"[2/3] renumber {len(broken)} of {len(chunked)} chunked items start at " f"{', '.join('-' + str(n) for n in starts)} instead of -1") else: - print(f"[2/3] renumber all {len(items)} chunked items already start at -1") + print(f"[2/3] renumber all {len(chunked)} chunked items already start at -1") errors: list[str] = [] notes: list[str] = [] fixed = 0 @@ -705,12 +744,12 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> if write: connection.commit() - for item in items: - if item.base_bytes != CHUNK_BYTES: - notes.append( - f"{item.base_path}: chunked but its base row is {item.base_bytes:,} bytes, not " - f"{CHUNK_BYTES:,} -- the app detects chunking by that exact length, so it will not reassemble" - ) + for item in coincidental: + notes.append( + f"{item.base_path}: has a numeric-suffixed sibling but its base row is " + f"{item.base_bytes:,} bytes, not {CHUNK_BYTES:,} -- treated as independent content and " + f"left alone, since the app only reassembles an item whose base row is exactly that long" + ) if fixed: print(f" {'renumbered' if write else 'would renumber'} to start at -1: {fixed}") return fixed, errors, notes @@ -892,7 +931,13 @@ def select(items: list[Item]) -> list[Item]: if result.status == "error": failed_items.append(result) elif result.status == "migrated" and write: - inserted, deleted = write_item(connection, item, result.slices, renumber=False) + try: + inserted, deleted = write_item(connection, item, result.slices, renumber=False) + except PathClash as clash: + failed_items.append(result) + errors.append(str(clash)) + result.slices = [] + continue inserted_total += inserted deleted_total += deleted wrote_migrated_content = True From 56d04c62c8b1342d6ccbcb12b316a770f6a097f4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 16:14:42 +0000 Subject: [PATCH 10/16] ADFA-5153: address review - append-only version log, gate continuation grouping, guard v2 declaration Three fixes from review: - declare_dictionary_version no longer DELETEs the version log. The table is append-only by contract (docs/documentation-database.md, DatabaseVersionResolver): the last-inserted row wins, so the INSERT alone declares version 2 and prior rows stay as history. Docstrings that asserted a one-row contract are corrected. - load_items only groups a "-" sibling as a continuation when its base row holds exactly CHUNK_BYTES, the app's own chunk-detection rule. Grouping on the name alone let phase 1's rewrite of a short base absorb an independent sibling page's bytes and delete its row. Every phase inherits the gate from this one choke point; phase_renumber's local copy of the test is now redundant and reduced to a comment. - Declaring version 2 over rows left plain is guarded two ways: --only-if-smaller is refused whenever the migrate phase runs (it deliberately leaves plain rows in a database that will declare 2, and a plain row can decode against the dictionary to wrong bytes without erroring), and a write run that declared 2 ends with an explicit WARNING when any brotli item did not migrate. --- .../migrate_content_to_dictionary_brotli.py | 76 +++++++++++++------ 1 file changed, 51 insertions(+), 25 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index f0c690413b..6c5cf624d8 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -334,15 +334,17 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: """ ).fetchall() - paths = {row[1] for row in rows} + base_bytes_by_path = {row[1]: row[5] for row in rows} items: dict[str, Item] = {} continuations: list[tuple[str, int, int, int]] = [] for row_id, path, language_id, type_id, template_id, length, type_value, compression in rows: match = CONTINUATION.match(path) - # A continuation only counts as one if its base is itself a row; a path - # that merely ends in - is ordinary content. - if match and match.group(1) in paths: + # A "-" sibling is a naming coincidence until the base row proves otherwise, and + # the proof is the base holding exactly CHUNK_BYTES -- the app's own chunk-detection rule. + # Grouping on the name alone let a rewrite of the greedy base (retype or migrate) absorb an + # independent page's bytes and delete its row; here every phase inherits the test. + if match and base_bytes_by_path.get(match.group(1)) == CHUNK_BYTES: continuations.append((match.group(1), row_id, int(match.group(2)), length)) else: items[path] = Item( @@ -501,8 +503,9 @@ def retype_rows(connection: sqlite3.Connection, item: Item, content_type_id: int def declared_major(connection: sqlite3.Connection) -> int | None: """The MAJOR this database declares, or None when it declares none. - Reads the row with the highest rowid: the table holds exactly one row by contract, and this is - the row both the app's DatabaseVersionResolver and docdb-studio read (ADFA-5220). + Reads the row with the highest rowid: the table is append-only by contract + (docs/documentation-database.md), so the row inserted last is the current version -- the same + row the app's DatabaseVersionResolver reads (ADFA-5220). """ if not table_exists(connection, "DocumentationDatabaseVersion"): return None @@ -515,12 +518,13 @@ def declared_major(connection: sqlite3.Connection) -> int | None: def declare_dictionary_version(connection: sqlite3.Connection) -> None: """Records that this database's brotli content is dictionary-compressed. - Written in the same transaction as the last batch of content, because the two facts have to - travel together: content compressed against the dictionary, and a version saying so. Exactly - one row, replaced rather than appended, matching populate_db.py. + Written in the same transaction as the first batch of migrated content, because the two facts + have to travel together: content compressed against the dictionary, and a version saying so. + The log is append-only by contract (docs/documentation-database.md, DatabaseVersionResolver): + each change is another INSERT and the row inserted last is the current version, so prior + version rows are history to keep, never state to replace. """ connection.execute(VERSION_TABLE_SQL) - connection.execute("DELETE FROM DocumentationDatabaseVersion") connection.execute( "INSERT INTO DocumentationDatabaseVersion (major, minor, patch, who, comment) VALUES (?, 0, 0, ?, ?)", ( @@ -715,14 +719,11 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> items = select([item for item in load_items(connection, "1 = 1") if item.continuations]) if args.renumber_scope == "retyped": items = [item for item in items if item.base_path in retyped_paths] - # A "-" sibling is a naming coincidence until the base row proves otherwise. The app - # decides an item is chunked by its base row being exactly CHUNK_BYTES (WebServer's continuation - # query), so that is the test here too. Without it this phase renamed real, independent pages: - # a page k/kotlin-1-2 whose greedy base is the real page k/kotlin-1 was renumbered to - # k/kotlin-1-1, which 404s every link to it and leaves the base looking like a 2-slice item. - # The check used to happen 20 lines below, as a note, after every rename had been made. - chunked = [item for item in items if item.base_bytes == CHUNK_BYTES] - coincidental = [item for item in items if item.base_bytes != CHUNK_BYTES] + # load_items only groups a "-" sibling under a base holding exactly CHUNK_BYTES -- the + # app's own chunk-detection rule -- so every item here is genuinely chunked, and a + # coincidentally named independent page (e.g. k/kotlin-1-2 next to the real page k/kotlin-1) + # never reaches the renames below. + chunked = items broken = [item for item in chunked if item.first_suffix != 1] starts = sorted({item.first_suffix for item in broken}) @@ -744,12 +745,6 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> if write: connection.commit() - for item in coincidental: - notes.append( - f"{item.base_path}: has a numeric-suffixed sibling but its base row is " - f"{item.base_bytes:,} bytes, not {CHUNK_BYTES:,} -- treated as independent content and " - f"left alone, since the app only reassembles an item whose base row is exactly that long" - ) if fixed: print(f" {'renumbered' if write else 'would renumber'} to start at -1: {fixed}") return fixed, errors, notes @@ -801,7 +796,8 @@ def main() -> int: parser.add_argument( "--only-if-smaller", action="store_true", - help="leave a row alone when its dictionary-compressed form is not smaller", + help="leave a row alone when its dictionary-compressed form is not smaller " + "(refused when the migrate phase runs: see the error it prints)", ) args = parser.parse_args() write = args.yes and not args.dry_run @@ -812,6 +808,20 @@ def main() -> int: print(f"error: unknown phase(s) {', '.join(unknown)}; pick from {', '.join(ALL_PHASES)}", file=sys.stderr) return 2 + # A migrate run declares version DICTIONARY_MAJOR_VERSION, and a plain-brotli row left behind + # in a database declaring that version can decode against the dictionary to different bytes + # *without erroring* -- served as silently wrong content. --only-if-smaller deliberately + # leaves such rows, so the two cannot travel together. + if args.only_if_smaller and "migrate" in args.phase_list: + print( + "error: --only-if-smaller cannot be combined with the migrate phase: it deliberately " + "leaves rows plain-compressed in a database the run declares version " + f"{DICTIONARY_MAJOR_VERSION}, and a plain row in such a database can decode against " + "the dictionary to wrong bytes without erroring", + file=sys.stderr, + ) + return 2 + connection = sqlite3.connect(args.database) connection.execute("PRAGMA foreign_keys = ON") @@ -1012,7 +1022,23 @@ def select(items: list[Item]) -> list[Item]: declare_dictionary_version(connection) print(f"declared database version {DICTIONARY_MAJOR_VERSION}.0.0 " f"(was {'none' if before is None else before})") + version_declared = True connection.commit() + # Attaching the dictionary to a plain-compressed row usually throws (the app then falls + # back to a plain decode), but a small fraction decode without error to different bytes. + # So a declared database still holding plain rows is not merely incomplete: it can serve + # silently wrong content. counts["error"] misses items that failed at the write (PathClash) + # after counting as migrated; failed_items holds both, so it is the honest tally. + remaining = counts["unchanged"] + len(failed_items) + if version_declared and remaining: + print( + f"\nWARNING: this database declares version {DICTIONARY_MAJOR_VERSION}.x but " + f"{remaining} brotli item(s) did not migrate and are still stored as before. " + f"A plain-compressed row can decode against the dictionary to wrong bytes " + f"without erroring, so re-run this script to completion before shipping " + f"this database.", + file=sys.stderr, + ) print("\nRun VACUUM to reclaim the freed pages: sqlite3 %s 'VACUUM;'" % args.database) else: connection.rollback() From 9914db6a6c2a9fd1047df1b7b190e28314030828 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Wed, 26 Aug 2026 15:14:13 -0700 Subject: [PATCH 11/16] ADFA-5153: A 1 MiB base is not proof of chunking The guard I added last round -- base row exactly CHUNK_BYTES -- rules out a small base and nothing else. A real page that happens to be exactly 1 MiB, sitting next to independently named "-2"/"-3" pages, was still grouped with them, and phase 2 renamed those pages into its slice slots. Reproduced against the real schema: p/page (1,048,576 bytes) plus p/page-2 and p/page-3 at 11 bytes each came out as p/page, p/page-1 (11 bytes), p/page-2 (11 bytes) -- both original URLs 404, one page gone, and the app appends a foreign page's bytes when it reassembles. The comment asserting this could not happen was wrong. The signal was already loaded: a genuine slice set has every slice except the last at exactly CHUNK_BYTES, because that is how the writer splits. An 11-byte "-2" followed by a "-3" is provably not one. The test now runs in load_items, so all three phases inherit it rather than phase 2 alone, and a sibling set that fails it becomes independent items instead of being silently absorbed. Verified: the two coincidence cases are left untouched, and both genuine mis-numbered slice sets are still repaired. A continuation whose base never became an Item was dropped silently -- not migrated, not counted, not reported, in a database the run then declares version 2. It is reported now. The version declaration is refused on a --path or --limit run. It covers the whole database, so only a run that considered the whole database may make it; a scoped run left tens of thousands of rows plain while telling the app they were dictionary-compressed. Most such rows throw and fall back, but a fraction decode without error to different bytes, which the code's own comment says two lines further down. The run now prints why it withheld the declaration. Phase 3 releases result.slices after writing, which phase 1 already did with a comment explaining why. Found in review of PR #1724. --- .../migrate_content_to_dictionary_brotli.py | 70 ++++++++++++++++++- 1 file changed, 68 insertions(+), 2 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 6c5cf624d8..2cf7a3a3ba 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -358,14 +358,54 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: base_bytes=length, ) + orphans: list[tuple[str, int, int, int]] = [] for base_path, row_id, suffix, length in continuations: owner = items.get(base_path) if owner is not None: owner.continuations.append((row_id, suffix, length)) + else: + # The greedy base is itself a continuation, or the phase predicate excluded it, so this + # row has no owner to be a slice of. Silently dropping it meant a row that was never + # migrated, never counted and never reported -- in a database the run then declares + # version 2. + orphans.append((base_path, row_id, suffix, length)) for item in items.values(): item.continuations.sort(key=lambda entry: entry[1]) + # A base of exactly CHUNK_BYTES is not enough on its own. A genuine slice set has every slice + # except the last at exactly CHUNK_BYTES, because that is how the writer splits; an 11-byte "-2" + # followed by a "-3" is provably not one. Without this, a real page that happens to be exactly + # 1 MiB, sitting next to independently named "-2"/"-3" pages, was grouped with them and phase 2 + # renamed those pages into its slice slots -- both URLs 404, and the app appends a foreign page's + # bytes on reassembly. Verified against the real schema before and after this check. + for item in list(items.values()): + if not item.continuations: + continue + head = item.continuations[:-1] + if all(length == CHUNK_BYTES for _, _, length in head): + continue + for row_id, suffix, length in item.continuations: + path = f"{item.base_path}-{suffix}" + items[path] = Item( + base_path=path, + base_id=row_id, + language_id=item.language_id, + content_type_id=item.content_type_id, + template_id=item.template_id, + content_type=item.content_type, + compression=item.compression, + base_bytes=length, + ) + item.continuations.clear() + + if orphans: + for base_path, _, suffix, _ in orphans: + print( + f" note: {base_path}-{suffix} looks like a continuation of {base_path}, which is " + f"not itself a migratable row; left alone and not migrated" + ) + return sorted(items.values(), key=lambda item: item.base_path) @@ -515,6 +555,23 @@ def declared_major(connection: sqlite3.Connection) -> int | None: return row[0] if row is not None and row[0] is not None else None +def may_declare_version(args) -> str: + """'' if this run may declare MAJOR 2, else why it may not. + + The declaration tells the app every brotli row is dictionary-compressed, and it applies to the + whole database -- so only a run that considered the whole database may make it. A --path or + --limit run migrates a handful and would leave the rest plain while claiming otherwise; the app + then attaches the dictionary to those rows, and while most throw and fall back, a fraction + decode without error to *different bytes*. That is silent wrong content, which is worse than the + unmigrated state it replaces. + """ + if args.path: + return f"the run was scoped by --path {args.path!r}" + if args.limit: + return f"the run was scoped by --limit {args.limit}" + return "" + + def declare_dictionary_version(connection: sqlite3.Connection) -> None: """Records that this database's brotli content is dictionary-compressed. @@ -951,6 +1008,10 @@ def select(items: list[Item]) -> list[Item]: inserted_total += inserted deleted_total += deleted wrote_migrated_content = True + # Same reason as phase 1: a completed Future holds its Result, and pending keeps + # every Future in the batch, so without this a batch of recompressed payloads + # stays resident while the next batch reads its own. + result.slices = [] if write: # In the same transaction as the first batch of migrated content, not after the last @@ -960,7 +1021,7 @@ def select(items: list[Item]) -> list[Item]: # would fail to decode. Declaring first means the worst an interruption leaves is a # partly migrated database that still serves, since the app falls back to a plain # decode per row. - if wrote_migrated_content and not version_declared: + if wrote_migrated_content and not version_declared and not may_declare_version(args): before = declared_major(connection) if before is None or before < DICTIONARY_MAJOR_VERSION: declare_dictionary_version(connection) @@ -1018,7 +1079,12 @@ def select(items: list[Item]) -> list[Item]: # Anything already dictionary-compressed still needs the declaration, even when this run # migrated nothing itself (every row came back "already"). before = declared_major(connection) - if not version_declared and (before is None or before < DICTIONARY_MAJOR_VERSION): + withheld = may_declare_version(args) + if withheld and (before is None or before < DICTIONARY_MAJOR_VERSION): + print(f"\nWARNING did NOT declare database version {DICTIONARY_MAJOR_VERSION}.0.0: {withheld}.") + print(" The declaration covers the whole database, so only an unscoped run may make") + print(" it. Re-run without --path/--limit before shipping this database.") + if not version_declared and not withheld and (before is None or before < DICTIONARY_MAJOR_VERSION): declare_dictionary_version(connection) print(f"declared database version {DICTIONARY_MAJOR_VERSION}.0.0 " f"(was {'none' if before is None else before})") From bbdb3ea5be9a59c12954a76a7b6a25e8a5398ae3 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 22:29:49 +0000 Subject: [PATCH 12/16] ADFA-5153: harden migration script against mid-run failures and bad flags Three findings from CodeRabbit's review of PR #1724, each verified against a scratch database before and after: - Guard future.result() in the retype and migrate as_completed loops. A worker exception (e.g. the brotli CLI vanishing mid-run) re-raised and aborted the phase with earlier batches already committed and no summary; it is now recorded as that item's error and the run completes. A shutil.which("brotli") preflight also refuses retype/migrate runs up front when the CLI is missing, next to the dictionary checks. - Validate --batch and --workers >= 1. Zero raised from range() or ProcessPoolExecutor after work may have started; a negative batch silently processed nothing. Both now exit 2 with a clear error, alongside the existing phase-name check. - verify_retype passed read_blobs() straight to b"".join(), so a row deleted or NULLed between the write and the verify aborted verification with a TypeError. It now records the problem and continues, matching the adjacent vanished-row branch. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0197g8vkUQ1d6oLNbi8EnAYe --- .../migrate_content_to_dictionary_brotli.py | 43 +++++++++++++++++-- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 2cf7a3a3ba..3e5bbdc7b8 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -75,6 +75,7 @@ import concurrent.futures as futures import os import re +import shutil import sqlite3 import subprocess import sys @@ -694,7 +695,13 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s for future in futures.as_completed(pending): item = pending[future] - found = future.result() + try: + found = future.result() + except Exception as exc: + # A worker crash used to re-raise here and abort the phase with earlier + # batches already committed and no summary. Report it like any other failure. + errors.append(f"{item.base_path}: {type(exc).__name__}: {exc}") + continue if found.status == "error": errors.append(f"{item.base_path}: {found.detail}") continue @@ -816,7 +823,11 @@ def verify_retype(connection, retyped_paths: set[str], mov_type: str, expect_ren if item is None: problems.append(f"{path}: row vanished") continue - payload = b"".join(read_blobs(connection, item)) + blobs = read_blobs(connection, item) + if blobs is None: + problems.append(f"{path}: a row is missing or holds NULL content") + continue + payload = b"".join(blobs) found = sniff(payload) expected = item.content_type if expected == "video/mp4" and mov_type == "mp4" and found == "video/quicktime": @@ -879,6 +890,15 @@ def main() -> int: ) return 2 + # range() raises on a zero batch, a negative one silently processes nothing, and + # ProcessPoolExecutor raises on zero workers -- all after work may have started. + if args.batch < 1: + print(f"error: --batch must be at least 1, got {args.batch}", file=sys.stderr) + return 2 + if args.workers < 1: + print(f"error: --workers must be at least 1, got {args.workers}", file=sys.stderr) + return 2 + connection = sqlite3.connect(args.database) connection.execute("PRAGMA foreign_keys = ON") @@ -887,6 +907,14 @@ def main() -> int: # numbering most needs repairing. dictionary = b"" if any(phase in ("retype", "migrate") for phase in args.phase_list): + # Fail before any phase runs, not per item inside a worker mid-run. + if shutil.which("brotli") is None: + print( + "error: retype and migrate need the brotli CLI (>= 1.0) on PATH; " + "no Python binding exposes custom dictionaries", + file=sys.stderr, + ) + return 2 if not table_exists(connection, "CompressionDictionary"): print( "error: this database has no CompressionDictionary table, so there is nothing to " @@ -988,7 +1016,16 @@ def select(items: list[Item]) -> list[Item]: for future in futures.as_completed(pending): item = pending[future] - result = future.result() + try: + result = future.result() + except Exception as exc: + # Same reason as the read_blobs guard above: report, do not abort a run + # whose earlier batches are already committed. + counts["error"] += 1 + failed_items.append( + Result(item.base_path, "error", detail=f"{type(exc).__name__}: {exc}") + ) + continue counts[result.status] += 1 before_total += result.before after_total += result.after or result.before From ca5f7e3b16cd64b3592b4d73d1e22c4531c51166 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 22:48:42 +0000 Subject: [PATCH 13/16] ADFA-5153: ungrouped rows keep their own metadata; renumber sets languageID Three fixes in load_items/renumber_item: - A row split back out of a disproved continuation group was built from the base item's languageID/contentTypeID/templateId, so phase 3 relabelled an independent page with the base's MIME type. It now carries its own row's metadata. - Orphaned "-N" rows (whose would-be base is itself a continuation) were logged and dropped, so a plain-brotli row silently survived a run that declares version 2. They now migrate as standalone items; a genuine stray slice fails to decode and feeds the existing "did not migrate" warning instead of vanishing. - renumber_item's UPDATE moved only the path. WebServer loads continuations with "languageId = 1" hardcoded, so a renumbered row under another language stayed invisible and the page still truncated. The UPDATE now normalises languageID like write_item does. Verified on scratch databases: split rows keep type/language/template through a migrate run; a migratable orphan round-trips against the dictionary while an undecodable one exits 1 with the version warning; renumbered continuations end at languageID 1. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0197g8vkUQ1d6oLNbi8EnAYe --- .../migrate_content_to_dictionary_brotli.py | 78 ++++++++++--------- 1 file changed, 43 insertions(+), 35 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index 3e5bbdc7b8..e7e750d5df 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -336,10 +336,28 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: ).fetchall() base_bytes_by_path = {row[1]: row[5] for row in rows} + rows_by_id = {row[0]: row for row in rows} items: dict[str, Item] = {} continuations: list[tuple[str, int, int, int]] = [] - for row_id, path, language_id, type_id, template_id, length, type_value, compression in rows: + def standalone(row: tuple) -> Item: + """An Item carrying the row's own metadata -- a row ungrouped later (a disproved + continuation, an orphan) is an independent page, and stamping the would-be base's + contentTypeID/languageID onto it would relabel a foreign page.""" + row_id, path, language_id, type_id, template_id, length, type_value, compression = row + return Item( + base_path=path, + base_id=row_id, + language_id=language_id, + content_type_id=type_id, + template_id=template_id, + content_type=type_value, + compression=compression, + base_bytes=length, + ) + + for row in rows: + row_id, path, length = row[0], row[1], row[5] match = CONTINUATION.match(path) # A "-" sibling is a naming coincidence until the base row proves otherwise, and # the proof is the base holding exactly CHUNK_BYTES -- the app's own chunk-detection rule. @@ -348,16 +366,7 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: if match and base_bytes_by_path.get(match.group(1)) == CHUNK_BYTES: continuations.append((match.group(1), row_id, int(match.group(2)), length)) else: - items[path] = Item( - base_path=path, - base_id=row_id, - language_id=language_id, - content_type_id=type_id, - template_id=template_id, - content_type=type_value, - compression=compression, - base_bytes=length, - ) + items[path] = standalone(row) orphans: list[tuple[str, int, int, int]] = [] for base_path, row_id, suffix, length in continuations: @@ -365,10 +374,10 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: if owner is not None: owner.continuations.append((row_id, suffix, length)) else: - # The greedy base is itself a continuation, or the phase predicate excluded it, so this - # row has no owner to be a slice of. Silently dropping it meant a row that was never - # migrated, never counted and never reported -- in a database the run then declares - # version 2. + # The greedy base is itself a continuation, so this row has no owner to be a slice + # of. Silently dropping it meant a row that was never migrated, never counted and + # never reported -- in a database the run then declares version 2 while the row is + # still plain brotli, which can decode against the dictionary to wrong bytes. orphans.append((base_path, row_id, suffix, length)) for item in items.values(): @@ -386,26 +395,22 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: head = item.continuations[:-1] if all(length == CHUNK_BYTES for _, _, length in head): continue - for row_id, suffix, length in item.continuations: - path = f"{item.base_path}-{suffix}" - items[path] = Item( - base_path=path, - base_id=row_id, - language_id=item.language_id, - content_type_id=item.content_type_id, - template_id=item.template_id, - content_type=item.content_type, - compression=item.compression, - base_bytes=length, - ) + for row_id, _, _ in item.continuations: + row = rows_by_id[row_id] + items[row[1]] = standalone(row) item.continuations.clear() - if orphans: - for base_path, _, suffix, _ in orphans: - print( - f" note: {base_path}-{suffix} looks like a continuation of {base_path}, which is " - f"not itself a migratable row; left alone and not migrated" - ) + # An orphan is still a row this phase selected, so it becomes its own item and migrates + # normally. If it really is a stray slice of some stream, its bytes decode neither plainly + # nor with the dictionary, and it surfaces as an error instead of silently surviving a run + # that declares version 2. + for base_path, row_id, suffix, _ in orphans: + row = rows_by_id[row_id] + items[row[1]] = standalone(row) + print( + f" note: {base_path}-{suffix} looks like a continuation of {base_path}, which is " + f"not itself a migratable row; treated as an independent page" + ) return sorted(items.values(), key=lambda item: item.base_path) @@ -636,8 +641,11 @@ def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> st # fail on UNIQUE(path) after the collision checks above had passed. for row_id, suffix, _ in sorted(item.continuations, key=lambda entry: entry[1]): connection.execute( - "UPDATE Content SET path = ? WHERE id = ?", - (f"{item.base_path}-{suffix - shift}", row_id), + # languageID too, for the same reason write_item normalises it: WebServer loads + # continuations with "languageId = 1" hardcoded, so a renumbered row left under + # another language is invisible and the page still truncates at its first 1 MiB. + "UPDATE Content SET path = ?, languageID = ? WHERE id = ?", + (f"{item.base_path}-{suffix - shift}", CONTINUATION_LANGUAGE_ID, row_id), ) return "" From 7a97936c2c421f2e8a415bdcfafd73a6ae5fc342 Mon Sep 17 00:00:00 2001 From: David Schachter Date: Mon, 31 Aug 2026 14:33:14 -0700 Subject: [PATCH 14/16] ADFA-5153: Fix 15 review findings in the dictionary migration script The theme is the version declaration and the run's report disagreeing with what is on disk. Six paths ended with the database declaring MAJOR 2, or committing recompressed rows, while rows were still plain -- and exiting 0. Correctness: - A --path/--limit --yes migrate committed recompressed rows but withheld the declaration, so the app could not decode them at all and the plaintext was gone. Refused up front now, like --only-if-smaller. A scoped dry run still runs. - No try/finally around the pool or connection, so Ctrl-C -- how a multi-hour run most often ends -- escaped past both end-of-run warnings, the summary and the exit code. Reproduced: SIGINT left 2.0.0 declared, 404 rows plain, a raw traceback and nothing tying the two together. Now aborts report and exit 1, and the "declares 2 but N did not migrate" warning counts unattempted rows. - The "already migrated" test decoded the STORED stream, but the question is about the RE-ENCODED one: rows stored at q1 were skipped permanently and counted as success. Measured on 30 such rows: was "migrated 0 / already 30 / saved 0 B", now "migrated 30 / saved 83.1%". - Both decodes can succeed and disagree, neither erroring -- a 92-byte stored stream decodes plainly to its real 49,200 bytes and, with the dictionary, to 49,200 bytes of garbage with exit 0. Recompressing the wrong one stored the garbage and the round-trip check validated it. The version the database declared BEFORE the run settles it, snapshotted so mid-run declaration cannot flip the answer. - all(...) over continuations[:-1] is vacuously true for a single sibling, so an unrelated 11-byte k/guide-2 was grouped under a real 1 MiB k/guide and renamed into its slice slot. Slices share the base's contentTypeID and languageID; siblings that disagree are left alone and reported. - Sibling detection read only the phase's selection, so a continuation typed outside it was invisible and could push a rewrite back to -2. Reads the whole table now and reports what it did not select. - renumber_item returned its success sentinel when shift <= 0, so a -0-based item was counted repaired untouched. Shifts up now, descending to avoid collisions. - Zero-padded "-007" parsed to 7 and rendered back as "-7", so the clash check probed a path no row held. Not treated as a continuation. - Continuations of an item already numbered from -1 never got the languageID normalisation, leaving them invisible to the app's chunk query. - Phase 1 and phase 3 both credited counts and byte totals before the write, so a PathClash was reported as success. Phase 1 emitted three errors for one failure. Robustness and reporting: - --phases "" ran nothing and exited 0 -- a CI step with an unset variable reported a successful migration of an untouched database. - --quality/--window unvalidated: a typo failed every item one subprocess at a time and still declared the version. --workers is capped at 64. - Preflight that Languages holds the id the continuation INSERT hardcodes. - The retype error told the operator to fix the shared ContentTypes row, which would break every row of that type -- every PDF in the Dynamic Bookshelf. - Dry runs never released recompressed payloads: 195,136 KB peak RSS against 117,888 KB, measured with only this line's indentation differing. Docs: the module docstring's incremental-safety claim and its "decodes identically" rule described the behaviour this commit replaces, and documentation-database.md asserted the dictionary-first/plain-fallback decode "correctly handles both dictionary-compressed and plain rows". It does not when the dictionary decode succeeds wrongly; corrected with the measurement. Verified: 11 targeted cases pass, 600 migrated rows all decode back to their original content, an interrupted run resumes to full integrity, and a re-run is a clean no-op. --- docs/documentation-database.md | 2 +- .../migrate_content_to_dictionary_brotli.py | 413 +++++++++++++++--- 2 files changed, 346 insertions(+), 69 deletions(-) diff --git a/docs/documentation-database.md b/docs/documentation-database.md index b8c6cd20ec..920f6ece2a 100644 --- a/docs/documentation-database.md +++ b/docs/documentation-database.md @@ -34,7 +34,7 @@ CREATE TABLE Content ( One row per file the web server can serve (HTML, CSS, JS, image, video, PDF, ...) — 30,000+ rows. Key points: - **`path`** is the lookup key (indexed via the `UNIQUE` constraint) and is what `WebServer` matches the HTTP request path against. Paths carry a short source prefix to avoid collisions between doc sets, e.g. `k/index.html` (Kotlin) vs `j/index.html` (Java). -- **`content`** is compressed — Brotli for text-like formats, format-specific compression otherwise (images/video/fonts). `ContentTypes.compression` says which. Every migrated `Content` row with `ContentTypes.compression = 'brotli'` is Brotli-compressed against the single shared dictionary in `CompressionDictionary` (see below), converted in one pass by ADFA-5153 — but plugin-contributed Tier 3 rows (`PluginDocumentationManager`/`BrotliCompressor`, see below) are plain, dictionary-free Brotli, and there is no per-row flag distinguishing the two, because a dictionary-compressed stream and a plain one are not distinguishable at decode time by inspection. They *are* distinguishable by attempting the decode: attaching the *wrong* dictionary decodes without error to different bytes than were compressed (its backward distances resolve into real, just incorrect, bytes) — but attaching *no* dictionary to a stream that needs one reliably throws (`IOException`, "corrupted input"), since distances into the dictionary region are then out of bounds for any spec-compliant decoder. `DocumentationContentSource` (which both Tier 3 transports read through) relies on exactly this: its decode tries the dictionary first and falls back to a plain decode on `IOException`, which correctly handles both dictionary-compressed and plain rows — but never rely on decode success/failure to detect a *wrong* dictionary, since that case is silent. Content over 1 MB is split across multiple rows: the first row's path is the base path, continuation rows are `path-1`, `path-2`, ... (`languageId = 1`), reassembled by `DocumentationContentSource` before returning. +- **`content`** is compressed — Brotli for text-like formats, format-specific compression otherwise (images/video/fonts). `ContentTypes.compression` says which. Every migrated `Content` row with `ContentTypes.compression = 'brotli'` is Brotli-compressed against the single shared dictionary in `CompressionDictionary` (see below), converted in one pass by ADFA-5153 — but plugin-contributed Tier 3 rows (`PluginDocumentationManager`/`BrotliCompressor`, see below) are plain, dictionary-free Brotli, and there is no per-row flag distinguishing the two, because a dictionary-compressed stream and a plain one are not distinguishable at decode time by inspection. They *are* distinguishable by attempting the decode: attaching the *wrong* dictionary decodes without error to different bytes than were compressed (its backward distances resolve into real, just incorrect, bytes) — but attaching *no* dictionary to a stream that needs one reliably throws (`IOException`, "corrupted input"), since distances into the dictionary region are then out of bounds for any spec-compliant decoder. `DocumentationContentSource` (which both Tier 3 transports read through) relies on exactly this: its decode tries the dictionary first and falls back to a plain decode on `IOException`. That handles the common case, **but the fallback is not a guarantee in either direction**: attaching a dictionary to a stream that never used one can also decode *without error, to different bytes*, in which case nothing throws and the fallback never fires. Measured with brotli 1.2.0: a 92-byte stored stream of repetitive HTML decodes plainly to its real 49,200 bytes and, with a dictionary attached, to 49,200 bytes of garbage with exit 0. Short stored streams are where this lives. So never rely on decode success/failure to detect a wrong dictionary *or* a dictionary/plain mismatch — both are silent — and treat a half-migrated database as able to serve wrong content, not merely as incomplete. Content over 1 MB is split across multiple rows: the first row's path is the base path, continuation rows are `path-1`, `path-2`, ... (`languageId = 1`), reassembled by `DocumentationContentSource` before returning. - **`templateId`**: `0` (or unset) means `content` is legacy HTML with presentation baked in (the pre-CMS Release 0/1 format). A positive value means `content` is JSON *facts only*, rendered through the matching row in `Templates` (a Pebble template) — the ongoing move to a proper CMS that de-duplicates presentation across near-identical pages (e.g. `sin`/`cos` docs). - Two data defects live in the shipped rows rather than in the schema, and `scripts/docdb/migrate_content_to_dictionary_brotli.py` repairs both before it recompresses anything. **Chunk numbering:** 14 of the 19 chunked items number their continuations from `-2`, not the `-1` the reassembly loop starts at (ADFA-5171), so those items serve as their first 1 MiB and nothing more; the script's `renumber` phase shifts them down. **Mislabelled types:** 74 rows holding GIF/PNG/JPEG/QuickTime payloads are typed `text/plain` (ADFA-5221), so they are Brotli-compressed for no gain and served as `Content-Type: text/plain`; the `retype` phase stores their plaintext and points them at the type their magic bytes prove they are, which -- since those types carry `compression = 'none'` -- also drops them out of the dictionary pass. Both defects originate in `docdb-studio`'s import path, so a freshly exported database will carry them again until fixed there. - The `UNIQUE(path)` constraint rejects any duplicate `path`, regardless of `languageID` — a second language for an existing path isn't supported yet (only `EN-us` currently exists). Getting there needs an upstream schema change to composite uniqueness on `(path, languageID)` (see *Known rough edges* below). diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index e7e750d5df..fa32c8b4e6 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -16,8 +16,15 @@ 3. migrate -- rewrite every `ContentTypes.compression = 'brotli'` row so it is compressed against the database's own dictionary rather than plainly. WebServer tries a dictionary-attached decode first and - falls back to a plain one, so a half-migrated database still - serves -- which is what makes running this incrementally safe. + falls back to a plain one, so a half-migrated database mostly + still serves -- but only mostly: a plain row can decode against + the dictionary to *different bytes without erroring*, so the + fallback never fires and the page serves silent garbage. A run + that declares the version therefore has to finish. The script + says so at the end when it did not, refuses --only-if-smaller + alongside migrate, and refuses a --path/--limit --yes migrate + outright, since a scoped run may not make the declaration and + content migrated without it cannot be decoded at all. Phase 1 feeds phase 3 for free: a row retyped to `image/gif` inherits that type's `compression = 'none'`, so phase 3's `compression = 'brotli'` selection simply @@ -32,10 +39,17 @@ a base row plus its continuations -- concatenated, decoded, rewritten, and re-split. Treating such rows one at a time would destroy the content. - * A few rows decode identically with and without the dictionary: tiny, - already-compressed payloads where the compressor found nothing to reference. - Those are left alone, so "already migrated" covers them as well as genuinely - dictionary-bound rows, and re-running does not churn them. + * Neither decode classifies a row on its own. Some rows decode identically with + and without the dictionary (tiny payloads the compressor found nothing to + reference in); others decode *both* ways to different bytes, neither erroring + -- a 92-byte stored stream of repetitive HTML decodes plainly to the real + content and, with the dictionary attached, to the same length of garbage. + What settles those is the version the database declared before the run + started: below the dictionary version no row is dictionary-compressed, so the + plain decode is the content; at or above it, the dictionary decode is. + Whether there is anything to *do* is then a separate question, decided by + re-encoding and comparing against the stored bytes -- not by how the stored + bytes happen to decode, which says nothing about what re-encoding would gain. * Extensions nominate phase 1's candidates; magic bytes decide. A row is retyped to what its payload actually is, not to what its name suggests, and a @@ -85,10 +99,25 @@ CHUNK_BYTES = 1024 * 1024 +# Each worker holds ~3 copies of an item's plaintext while it works; the largest chunked items in +# this database are ~160 MB, so the ceiling is memory, not cores. +MAX_WORKERS = 64 + # WebServer looks continuations up with "languageId = 1" hardcoded, whatever the base row says. CONTINUATION_LANGUAGE_ID = 1 +# Every phase calls load_items, and verify_retype calls it again, so a note about the shape of the +# data would otherwise print three or four times per run for the same row. +_REPORTED: set[str] = set() + + +def note_once(message: str) -> None: + if message not in _REPORTED: + _REPORTED.add(message) + print(message) + + def is_text_type(value: str) -> bool: """Whether a ContentTypes.value is a text type, matched at the boundary. @@ -97,7 +126,11 @@ def is_text_type(value: str) -> bool: app's ContentTypeHeaders (ADFA-5241). """ return value == "text" or value.startswith("text/") -CONTINUATION = re.compile(r"^(.*)-(\d+)$") +# The suffix must render back to the path it was parsed from. A zero-padded "-007" parses to 7 +# and renders as "-7", so the clash check probes a path no row holds while write_item UPDATEs the +# row still named "-007" and INSERTs the next slice at "-8" -- one item split across two naming +# schemes, reported as a success. A padded sibling is not one of ours; leave it standalone. +CONTINUATION = re.compile(r"^(.*)-(0|[1-9]\d*)$") ALL_PHASES = ("retype", "renumber", "migrate") # Extensions worth a second look when a row claims to be text. The extension only @@ -269,37 +302,44 @@ def inspect_item(item: Item, blobs: list[bytes]) -> Inspection: before=before, after=len(payload)) -def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only_if_smaller: bool) -> Result: +def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only_if_smaller: bool, + db_was_migrated: bool) -> Result: """Decode an item, recompress it against the dictionary, and re-split it. - Classification deliberately tries the *plain* decode first. Attaching no - dictionary to a stream that needs one reliably fails, so a successful plain - decode proves the row is not yet migrated; the reverse test is not safe, - because a dictionary attached to a stream that never used one can decode to - different bytes without erroring. + Both decodes are attempted, because neither one alone classifies a row. Attaching a + dictionary to a stream that never used one *usually* throws -- but not always, and the same + is true the other way, so a decode that succeeds is evidence and not proof. [db_was_migrated] + is what the database declared BEFORE this run started, and it settles the case where both + decodes succeed and disagree. """ stored = b"".join(blobs) before = len(stored) ok, plaintext = decode_plain(stored) + ok_dict, as_dict = decode_with_dictionary(stored) + if not ok: - ok_dict, _ = decode_with_dictionary(stored) if ok_dict: return Result(item.base_path, "already", before=before, after=before) return Result(item.base_path, "error", before=before, detail="decodes neither plainly nor with the dictionary") + # Both decodes succeeded and returned different bytes, neither erroring. This is not exotic: + # a short stored stream (92 B of highly repetitive HTML, say) decodes plainly to the real + # content and, with the dictionary attached, to the same LENGTH of garbage with exit 0. + # Nothing in the stream says which is which, so the decision comes from the database rather + # than the row: the version it declared before this run started. Below the dictionary version, + # no row is dictionary-compressed by contract, so the plain decode is the content. At or above + # it, they all are, so the dictionary decode is -- and the row needs nothing done to it. + # Recompressing the wrong one of these stores garbage that the round-trip check below then + # happily validates, because it round-trips the garbage. + if ok_dict and as_dict != plaintext: + if db_was_migrated: + return Result(item.base_path, "already", before=before, after=before) + # Decoding every row here anyway makes a mislabel sweep free: report a # text-typed row whose payload is recognisably binary, whatever its name. mislabelled = sniff(plaintext) if is_text_type(item.content_type) else "" - # A stream that decodes *both* ways is one the compressor never referenced the - # dictionary for -- small, already-compressed payloads like a 1 KB GIF. It is - # byte-identical in either form, so there is nothing to migrate, and skipping it - # keeps a re-run from recompressing it for no gain. - ok_dict, as_dict = decode_with_dictionary(stored) - if ok_dict and as_dict == plaintext: - return Result(item.base_path, "already", before=before, after=before, sniffed=mislabelled) - ok, recompressed, stderr = encode_with_dictionary(plaintext, quality, window) if not ok: return Result(item.base_path, "error", before=before, detail=f"compression failed: {stderr}") @@ -314,6 +354,16 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only detail="recompressed bytes do not decode back to the original content", ) + # "Nothing to do" is a property of the RE-ENCODED stream, not the stored one. The test this + # replaces decoded the stored stream twice and called any row that decoded both ways "already + # migrated" -- but a row stored at q1 also decodes identically with the dictionary attached, + # while re-encoding it at q11 against the dictionary is 36% smaller (5,858 -> 3,748 B on a real + # doc page). Every such row was skipped permanently and counted as a success: a whole-database + # run over content stored at q1 reported "already 300, saved 0 B" and exited 0 having done + # nothing. Comparing the bytes we would actually write is the question being asked. + if recompressed == stored: + return Result(item.base_path, "already", before=before, after=before, sniffed=mislabelled) + if only_if_smaller and len(recompressed) >= before: return Result(item.base_path, "unchanged", before=before, after=before, sniffed=mislabelled, detail=f"dictionary-compressed form is larger ({len(recompressed)} vs {before})") @@ -335,7 +385,21 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: """ ).fetchall() - base_bytes_by_path = {row[1]: row[5] for row in rows} + # Sibling detection reads the WHOLE table, not just this phase's selection. Scoping it to the + # predicate made a continuation typed outside the phase invisible: the item loaded with the + # wrong slice set, the unselected row survived unmigrated and unreported, and a base whose + # "-1" was filtered out reported first_suffix 2, so write_item re-created the stream starting + # at -2 -- re-introducing the ADFA-5171 defect phase 2 had just normalised. + all_rows = connection.execute( + "SELECT id, path, languageID, contentTypeID, IFNULL(LENGTH(content), 0) FROM Content" + ).fetchall() + base_bytes_by_path = {path: length for _, path, _, _, length in all_rows} + selected_ids = {row[0] for row in rows} + unselected_by_id = { + row_id: (path, language_id, type_id) + for row_id, path, language_id, type_id, _ in all_rows + if row_id not in selected_ids + } rows_by_id = {row[0]: row for row in rows} items: dict[str, Item] = {} continuations: list[tuple[str, int, int, int]] = [] @@ -389,17 +453,67 @@ def standalone(row: tuple) -> Item: # 1 MiB, sitting next to independently named "-2"/"-3" pages, was grouped with them and phase 2 # renamed those pages into its slice slots -- both URLs 404, and the app appends a foreign page's # bytes on reassembly. Verified against the real schema before and after this check. + def proven_slices(item: Item) -> str: + """'' when these really are slices of `item`, else why they cannot be proven to be.""" + head = item.continuations[:-1] + if not all(length == CHUNK_BYTES for _, _, length in head): + return "a slice before the last is not exactly CHUNK_BYTES" + # `head` is empty when there is exactly one continuation, so the length rule above is + # vacuously true and proves nothing -- that is how an unrelated 11-byte "k/guide-2" got + # grouped under a real 1 MiB "k/guide" and renamed into its slice slot by phase 2, 404ing + # its own URL and corrupting the reassembly of the base. Slices of one payload are written + # by write_item with a single contentTypeID and languageID, so a sibling that disagrees + # with the base on either is somebody else's page. This cannot separate a coincidence that + # happens to match on both; phase 3's decode is the backstop for that. + for row_id, _, _ in item.continuations: + row = rows_by_id[row_id] + if row[3] != item.content_type_id: + return f"{row[1]} is contentTypeID {row[3]}, not the base's {item.content_type_id}" + if row[2] not in (CONTINUATION_LANGUAGE_ID, item.language_id): + return (f"{row[1]} is languageID {row[2]}, which is neither " + f"{CONTINUATION_LANGUAGE_ID} nor the base's {item.language_id}") + return "" + for item in list(items.values()): if not item.continuations: continue - head = item.continuations[:-1] - if all(length == CHUNK_BYTES for _, _, length in head): + unproven = proven_slices(item) + if not unproven: continue + # Reported, not silently dropped. A genuine slice set whose rows disagree with the base is + # indistinguishable here from an independent page that merely shares the name, and the two + # want opposite treatment -- normalise, or keep well away. Renaming the wrong one destroys a + # live page, so the safe branch is taken and the operator is told which item to look at. + note_once( + f" note: {item.base_path} has {len(item.continuations)} '-N' sibling(s) that cannot " + f"be proven to be its slices ({unproven}); treated as independent pages. If they really " + f"are slices, the page will serve only its first {human(CHUNK_BYTES)} until they are fixed by hand" + ) for row_id, _, _ in item.continuations: row = rows_by_id[row_id] items[row[1]] = standalone(row) item.continuations.clear() + # A sibling this phase did not select is still a row that exists. Left unmentioned it survives + # every run unmigrated while the database goes on to declare the dictionary version. Indexed + # by base path first: probing the whole unselected set per item is O(items x rows), and both + # are ~30k here. + unselected_slices: dict[str, list[int]] = {} + for path, _, _ in unselected_by_id.values(): + match = CONTINUATION.match(path) + if match: + unselected_slices.setdefault(match.group(1), []).append(int(match.group(2))) + + for item in items.values(): + if not item.continuations: + continue + owned = {suffix for _, suffix, _ in item.continuations} + for suffix in sorted(set(unselected_slices.get(item.base_path, ())) - owned): + note_once( + f" note: {item.base_path}-{suffix} names a slice of {item.base_path} but this " + f"phase did not select it; it is left as it is and does not count towards the migration" + ) + # An orphan is still a row this phase selected, so it becomes its own item and migrates # normally. If it really is a stray slice of some stream, its bytes decode neither plainly # nor with the dictionary, and it surfaces as an error instead of silently surviving a run @@ -407,7 +521,7 @@ def standalone(row: tuple) -> Item: for base_path, row_id, suffix, _ in orphans: row = rows_by_id[row_id] items[row[1]] = standalone(row) - print( + note_once( f" note: {base_path}-{suffix} looks like a continuation of {base_path}, which is " f"not itself a migratable row; treated as an independent page" ) @@ -614,9 +728,9 @@ def content_types(connection: sqlite3.Connection) -> dict[str, tuple[int, str]]: def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> str: - """Shift an item's continuations down so they start at -1. Returns a note, or ''.""" + """Shift an item's continuations onto -1. Returns a note, or ''.""" shift = item.first_suffix - 1 - if shift <= 0: + if shift == 0: return "" expected = list(range(item.first_suffix, item.first_suffix + len(item.continuations))) @@ -633,13 +747,15 @@ def renumber_item(connection: sqlite3.Connection, item: Item, write: bool) -> st return f"{target} already exists and belongs to another row; left alone" if write: - # One ascending pass, no temporary names. The suffixes were just verified - # contiguous, so the lowest target (first_suffix - shift) is free -- nothing - # occupies a suffix below first_suffix -- and every later target was vacated by - # the move before it. The parking pass this replaces invented - # "{base}-renumbering-{n}" paths that a real row could already hold, which would - # fail on UNIQUE(path) after the collision checks above had passed. - for row_id, suffix, _ in sorted(item.continuations, key=lambda entry: entry[1]): + # One pass, no temporary names. The suffixes were just verified contiguous, so when + # shifting DOWN the lowest target is free (nothing occupies a suffix below first_suffix) + # and every later target was vacated by the move before it -- hence ascending order. An + # item numbered from -0 shifts UP instead (shift < 0), where the same argument runs + # backwards: the highest target is free and each earlier move is vacated by the one after + # it, so the pass has to descend. Getting this direction wrong walks each row onto the one + # ahead of it and fails on UNIQUE(path) after the collision checks above have passed -- + # which is what the "{base}-renumbering-{n}" parking pass this replaces existed to avoid. + for row_id, suffix, _ in sorted(item.continuations, key=lambda entry: entry[1], reverse=shift < 0): connection.execute( # languageID too, for the same reason write_item normalises it: WebServer loads # continuations with "languageId = 1" hardcoded, so a renumbered row left under @@ -743,17 +859,18 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s # then reports twice -- and if WebServer does not handle that compression at all, the row # serves raw compressed bytes to a browser. if compression != "none": + # NOT "fix the ContentTypes row": that row is a shared dimension. In the shipped + # database application/pdf, application/wasm, font/otf and font/ttf are all + # registered brotli and all four extensions are in BINARY_EXTENSIONS, so following + # that advice would set a legitimately compressed type to 'none' and make every + # row of it -- every PDF in the Dynamic Bookshelf -- serve as raw compressed bytes. errors.append( f"{item.base_path}: {target} is registered with compression '{compression}', not 'none'; " - f"left as {item.content_type}. Fix the ContentTypes row, then re-run" + f"left as {item.content_type}. Retype this row by hand, or exclude it with --path; " + f"do not change the ContentTypes row, which every other row of that type shares" ) continue - counts[target] = counts.get(target, 0) + 1 - before_total += found.before - after_total += found.after - kept.add(item.base_path) - if write: try: inserted, deleted = write_item( @@ -765,6 +882,16 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s continue inserted_total += inserted deleted_total += deleted + + # Credited only once the write has actually happened. Crediting before it meant a row + # whose write raised PathClash was still reported retyped, still had its bytes booked + # into the plaintext totals, and was still handed to verify_retype -- which then read + # the untouched row and emitted two more errors describing a disagreement that did not + # exist, three errors for one failure. + counts[target] = counts.get(target, 0) + 1 + before_total += found.before + after_total += found.after + kept.add(item.base_path) # found.slices is the only large thing here; dropping the reference lets this batch's # plaintext be collected before the next batch decodes its own. found.slices = [] @@ -814,6 +941,40 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> errors.append(f"{item.base_path}: {note}") else: fixed += 1 + + # An item already numbered from -1 never reaches renumber_item, so it never got the languageID + # normalisation that lives there. WebServer's continuation query hardcodes "languageId = 1", so + # a slice left under another language is invisible and the page truncates at its first 1 MiB -- + # the ADFA-5171 symptom, in an item this phase would otherwise report as healthy and pass to a + # run that declares the dictionary version. + relanguaged = 0 + for item in chunked: + if item.first_suffix != 1 or not item.continuations: + continue + ids = [row_id for row_id, _, _ in item.continuations] + stray = [ + row_id + for (row_id,) in connection.execute( + f"SELECT id FROM Content WHERE id IN ({','.join('?' * len(ids))}) AND languageID != ?", + [*ids, CONTINUATION_LANGUAGE_ID], + ) + ] + if not stray: + continue + relanguaged += len(stray) + notes.append( + f"{item.base_path}: {len(stray)} continuation row(s) carried a languageID other than " + f"{CONTINUATION_LANGUAGE_ID}, which hides them from the app's chunk query" + ) + if write: + connection.execute( + f"UPDATE Content SET languageID = ? WHERE id IN ({','.join('?' * len(stray))})", + [CONTINUATION_LANGUAGE_ID, *stray], + ) + if relanguaged: + print(f" {'set' if write else 'would set'} languageID = {CONTINUATION_LANGUAGE_ID} on " + f"{relanguaged} continuation row(s) that were hidden from the app") + if write: connection.commit() @@ -865,7 +1026,8 @@ def main() -> int: help="renumber every -2-based chunked item, or only the ones phase 1 retyped (default: all)") parser.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 2)), help="parallel compressors") parser.add_argument("--quality", type=int, default=11, help="brotli quality (default 11, as the pipeline uses)") - parser.add_argument("--window", type=int, default=22, help="brotli window log (default 22, the portable maximum)") + parser.add_argument("--window", type=int, default=22, + help="brotli window log, 0 or 10-24 (default 22; the app's BrotliCompressor uses 24)") parser.add_argument("--limit", type=int, default=0, help="stop after this many items (for a smoke test)") parser.add_argument("--path", default="", help="only items whose base path contains this substring") parser.add_argument("--batch", type=int, default=200, help="items per write transaction") @@ -883,6 +1045,14 @@ def main() -> int: if unknown: print(f"error: unknown phase(s) {', '.join(unknown)}; pick from {', '.join(ALL_PHASES)}", file=sys.stderr) return 2 + # --phases "" and --phases "," both survived the comprehension as an empty list, whose + # unknown-phase check iterates nothing: the run printed "mode WRITING", skipped the brotli and + # CompressionDictionary preflight, executed no phase and exited 0. A CI step writing + # --phases "$PHASES" with the variable unset reported a successful migration of a database it + # had not touched, defeating the exit code this script goes to some trouble to make meaningful. + if not args.phase_list: + print(f"error: --phases selected no phases; pick from {', '.join(ALL_PHASES)}", file=sys.stderr) + return 2 # A migrate run declares version DICTIONARY_MAJOR_VERSION, and a plain-brotli row left behind # in a database declaring that version can decode against the dictionary to different bytes @@ -898,6 +1068,24 @@ def main() -> int: ) return 2 + # A run that may not declare the version may not write migrated content either -- the two + # travel together or not at all. --path/--limit scope the run, so may_declare_version withholds + # the declaration; phase 3 nevertheless recompressed those rows and COMMITTED them, printed a + # WARNING on stdout and exited 0. Because the app gates the dictionary on the declared version + # and not on the dictionary's presence, every row just rewritten decodes as "corrupt input", + # and the plaintext it was rewritten from is gone. Refused here, like --only-if-smaller above, + # rather than warned about after the damage. A scoped dry run is still useful and still allowed. + withheld = may_declare_version(args) + if write and "migrate" in args.phase_list and withheld: + print( + f"error: {withheld}, so this run may not declare database version " + f"{DICTIONARY_MAJOR_VERSION} -- and content migrated without that declaration cannot be " + f"decoded by the app at all. Re-run without --path/--limit to migrate, or drop --yes to " + f"see what a scoped run would do.", + file=sys.stderr, + ) + return 2 + # range() raises on a zero batch, a negative one silently processes nothing, and # ProcessPoolExecutor raises on zero workers -- all after work may have started. if args.batch < 1: @@ -906,6 +1094,24 @@ def main() -> int: if args.workers < 1: print(f"error: --workers must be at least 1, got {args.workers}", file=sys.stderr) return 2 + # Same reason as the two above, and the same one-line shape: the brotli CLI rejects these, but + # only per row inside a worker, so a typo spent a whole pass failing every item one subprocess + # at a time -- and the run still declared the dictionary version at the end, leaving a database + # claiming every brotli row was migrated when not one had been. + if not 0 <= args.quality <= 11: + print(f"error: --quality must be between 0 and 11, got {args.quality}", file=sys.stderr) + return 2 + if args.window != 0 and not 10 <= args.window <= 24: + print(f"error: --window must be 0 or between 10 and 24, got {args.window}", file=sys.stderr) + return 2 + # Each worker holds roughly three copies of an item's plaintext (the decode, the dictionary + # decode and the round-trip), and the largest chunked items here reach ~160 MB, so a high + # worker count is an out-of-memory risk rather than a throughput win. + if args.workers > MAX_WORKERS: + print(f"error: --workers above {MAX_WORKERS} risks running the machine out of memory " + f"(~3x an item's plaintext per worker, and the largest items here are ~160 MB), " + f"got {args.workers}", file=sys.stderr) + return 2 connection = sqlite3.connect(args.database) connection.execute("PRAGMA foreign_keys = ON") @@ -936,6 +1142,23 @@ def main() -> int: return 2 dictionary = dictionary_row[0] + # write_item INSERTs continuations with languageID = CONTINUATION_LANGUAGE_ID, under + # PRAGMA foreign_keys = ON. Its call sites catch PathClash only, so on a database that numbers + # its languages differently the FK violation propagated out of main() from the middle of a + # phase, with earlier batches already committed and no summary printed. One query, up front. + if write and table_exists(connection, "Languages"): + if connection.execute( + "SELECT 1 FROM Languages WHERE id = ?", (CONTINUATION_LANGUAGE_ID,) + ).fetchone() is None: + print( + f"error: this database's Languages table has no id {CONTINUATION_LANGUAGE_ID}, but the " + f"app's continuation query hardcodes that id, so every slice this script writes would " + f"be invisible (and rejected by the foreign key). Nothing written.", + file=sys.stderr, + ) + connection.close() + return 2 + def select(items: list[Item]) -> list[Item]: if args.path: items = [item for item in items if args.path in item.base_path] @@ -953,7 +1176,25 @@ def select(items: list[Item]) -> list[Item]: retyped_paths: set[str] = set() started = time.time() - with futures.ProcessPoolExecutor(args.workers, initializer=_init_worker, initargs=(dictionary,)) as pool: + # Phase 3's tallies live out here, not next to its loop, because the summary below has to be + # printable whether or not that loop ran or finished -- see the abort handling after the try. + counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} + wrote_migrated_content = False + version_declared = False + migrate_started = False + before_total = after_total = 0 + inserted_total = deleted_total = 0 + # Not named `errors`: that name already holds this run's phase 1 and 2 failures, + # and reusing it here would discard them. + failed_items: list[Result] = [] + mislabelled: list[Result] = [] + aborted = "" + + # Not a `with`: on the way out of one, ProcessPoolExecutor.shutdown waits for every future it + # has already queued, so a Ctrl-C hung on the work it was trying to abandon. Built explicitly + # so the finally can cancel instead. + pool = futures.ProcessPoolExecutor(args.workers, initializer=_init_worker, initargs=(dictionary,)) + try: if "retype" in args.phase_list: candidates = select([ item for item in load_items(connection, "(CT.value = 'text' OR CT.value LIKE 'text/%')") @@ -993,15 +1234,13 @@ def select(items: list[Item]) -> list[Item]: print(f" stored now {human(sum(item.stored_bytes for item in items))}") print() - counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} - wrote_migrated_content = False - version_declared = False - before_total = after_total = 0 - inserted_total = deleted_total = 0 - # Not named `errors`: that name already holds this run's phase 1 and 2 failures, - # and reusing it here would discard them. - failed_items: list[Result] = [] - mislabelled: list[Result] = [] + migrate_started = True + + # Snapshotted BEFORE the first batch declares anything. This run declares the dictionary + # version as soon as it commits migrated content, so reading the declaration per item would + # flip the answer halfway through and start treating the still-plain rows as migrated. + starting_major = declared_major(connection) + db_was_migrated = starting_major is not None and starting_major >= DICTIONARY_MAJOR_VERSION for offset in range(0, len(items), args.batch): batch = items[offset : offset + args.batch] @@ -1018,7 +1257,8 @@ def select(items: list[Item]) -> list[Item]: continue pending[ pool.submit( - migrate_item, item, blobs, args.quality, args.window, args.only_if_smaller + migrate_item, item, blobs, args.quality, args.window, args.only_if_smaller, + db_was_migrated, ) ] = item @@ -1034,29 +1274,41 @@ def select(items: list[Item]) -> list[Item]: Result(item.base_path, "error", detail=f"{type(exc).__name__}: {exc}") ) continue - counts[result.status] += 1 - before_total += result.before - after_total += result.after or result.before - if result.sniffed: mislabelled.append(result) - if result.status == "error": - failed_items.append(result) - elif result.status == "migrated" and write: + + # The write can still fail, so nothing is credited until it has not. Counting + # first tallied a PathClash as "migrated" and subtracted its never-written bytes + # from the reported savings while counts["error"] stayed 0. + if result.status == "migrated" and write: try: inserted, deleted = write_item(connection, item, result.slices, renumber=False) - except PathClash as clash: + except (PathClash, sqlite3.IntegrityError) as clash: + counts["error"] += 1 + # PathClash already names the item, and the failed_items loop prefixes the + # path again when it prints. Strip it here, and report through failed_items + # alone -- appending to `errors` too printed the same failure twice. + detail, prefix = str(clash), f"{item.base_path}: " + result.detail = detail[len(prefix):] if detail.startswith(prefix) else detail failed_items.append(result) - errors.append(str(clash)) result.slices = [] continue inserted_total += inserted deleted_total += deleted wrote_migrated_content = True - # Same reason as phase 1: a completed Future holds its Result, and pending keeps - # every Future in the batch, so without this a batch of recompressed payloads - # stays resident while the next batch reads its own. - result.slices = [] + + counts[result.status] += 1 + before_total += result.before + after_total += result.after or result.before + if result.status == "error": + failed_items.append(result) + # Same reason as phase 1: a completed Future holds its Result, and pending keeps + # every Future in the batch, so without this a batch of recompressed payloads stays + # resident while the next batch reads its own. Outside the write branch it used to be + # indented into, because a dry run builds exactly the same payloads and never + # released them: 195,136 KB peak RSS against 117,888 KB, measured on 40 incompressible + # 2 MiB rows at --batch 40 with the only difference being this line's indentation. + result.slices = [] if write: # In the same transaction as the first batch of migrated content, not after the last @@ -1086,7 +1338,27 @@ def select(items: list[Item]) -> list[Item]: flush=True, ) + # An abort must not skip the summary. Every one of these used to escape main() as a traceback, + # past both end-of-run version warnings, past report() and past the exit code -- leaving a + # database that had already declared the dictionary version with most of its rows still plain + # brotli, and nothing on screen connecting the two. Ctrl-C is how a multi-hour run most often + # ends, so this is the common case, not the exotic one. + except KeyboardInterrupt: + aborted = "interrupted with Ctrl-C" + except futures.process.BrokenProcessPool: + aborted = "a worker process died (killed by the OOM killer, most likely)" + except sqlite3.Error as exc: + aborted = f"the database rejected an operation: {type(exc).__name__}: {exc}" + finally: + pool.shutdown(wait=False, cancel_futures=True) + print("\n") + if aborted: + print(f"ABORTED {aborted}") + print(" Committed batches are kept; what follows describes the work that") + print(" had finished when the run stopped.") + if aborted and not migrate_started: + print(" The migrate phase had not started.") print(f"migrated {counts['migrated']:,}") print(f"already {counts['already']:,}") if counts["unchanged"]: @@ -1140,7 +1412,12 @@ def select(items: list[Item]) -> list[Item]: # So a declared database still holding plain rows is not merely incomplete: it can serve # silently wrong content. counts["error"] misses items that failed at the write (PathClash) # after counting as migrated; failed_items holds both, so it is the honest tally. - remaining = counts["unchanged"] + len(failed_items) + # Rows the run never reached count too. An abort leaves most of them unattempted, and they + # are exactly as exposed as a row that failed: plain brotli in a database declaring the + # dictionary version. + attempted = sum(counts.values()) + unattempted = max(0, len(items) - attempted) if migrate_started else 0 + remaining = counts["unchanged"] + len(failed_items) + unattempted if version_declared and remaining: print( f"\nWARNING: this database declares version {DICTIONARY_MAJOR_VERSION}.x but " @@ -1161,7 +1438,7 @@ def select(items: list[Item]) -> list[Item]: print("\nNothing written. Re-run with --yes on a copy to apply.") connection.close() - return 1 if errors or failed_items else 0 + return 1 if aborted or errors or failed_items else 0 if __name__ == "__main__": From 9b12d063687aebfaeb0391fe1b5c9afd6fb563aa Mon Sep 17 00:00:00 2001 From: David Schachter Date: Mon, 31 Aug 2026 15:59:46 -0700 Subject: [PATCH 15/16] ADFA-5153: Clear the review's below-cap findings in the migration script Smaller than the 15 in the previous commit, but each one is real. - The two decode wrappers discarded brotli's stderr while encode_with_dictionary kept it, so every non-zero exit read as "this row does not decode": a vanished dictionary tempfile, an OOM-killed child or a missing binary were all reported as corruption in the content, sending the operator to the wrong place. Both decoders now return stderr and the errors quote it. - The mislabel census was filled in on only one of the branches that return "already", and not the one a dictionary-bound row takes -- so a re-run over an already-migrated database reported zero mislabelled rows regardless of how many it held. Verified on a dictionary-bound GIF typed text/plain: was silent, now reported. - LENGTH() on a TEXT-class value counts CHARACTERS, not bytes (5 vs 6 for "hello" with one two-byte character), and that number decides what counts as a 1 MiB chunk base. The app measures the same rows with getBlob().size. Both length queries CAST to BLOB now, as does read_blobs -- sqlite3 otherwise hands back str for a TEXT row, which fails in the worker at b"".join(blobs). Dead code, removed rather than documented: - --only-if-smaller: refused whenever migrate runs, and migrate_item was its only reader, so no invocation could reach it. Its "unchanged" status went with it. - --renumber-scope: phase 1 already passes renumber= to write_item, so a retyped item starts at -1 before phase 2 looks at it and `broken` never intersects retyped_paths. Always a no-op, whichever value was passed. - retype_rows(): never called, and it lacks write_item's clash and languageID guards, so reaching for it would reintroduce the bugs write_item exists to prevent. - phase_renumber's `notes` is no longer always empty -- the languageID normalisation added in the previous commit populates it. build.gradle.kts: the Spotless exclusion's comment claimed "every .py already here is space-indented". scripts/r8-plugin-impact/analyze-plugin-impact.py is tab-indented. The exclusion is still right; the reason given for it was not. Not done: the hard batch barrier's ~+92% wall clock. Fixing it means replacing the drain-then-commit structure with a sliding submission window, which is what currently bounds memory -- and three fixes in the previous commit depend on those bounds. Worth its own change, with its own measurements. Verified: the 11-case suite still passes, 600 rows round-trip, and renumbering is byte-for-byte unchanged by the removals. --- build.gradle.kts | 5 +- .../migrate_content_to_dictionary_brotli.py | 130 ++++++++---------- 2 files changed, 59 insertions(+), 76 deletions(-) diff --git a/build.gradle.kts b/build.gradle.kts index b4c56f6bab..efd9efb2cf 100755 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -338,8 +338,9 @@ spotless { targetExclude( "scripts/debug-keystore/adfa-keystore.jks", // leadingSpacesToTabs() would reindent Python, which PEP 8 indents with spaces -- - // and every .py already here is space-indented. Only the ratchet has been hiding - // that mismatch: an edit to one of them would silently convert the whole file. + // as all but one .py here already is (scripts/r8-plugin-impact/analyze-plugin-impact.py + // uses tabs). Only the ratchet has been hiding the mismatch: an edit to a + // space-indented one would silently convert the whole file. "**/*.py", // Python bytecode: binary, generated, and Spotless fails the whole task (and so the // pre-push hook) on one stray file rather than skipping it. diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index fa32c8b4e6..f1ff597edd 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -21,10 +21,10 @@ the dictionary to *different bytes without erroring*, so the fallback never fires and the page serves silent garbage. A run that declares the version therefore has to finish. The script - says so at the end when it did not, refuses --only-if-smaller - alongside migrate, and refuses a --path/--limit --yes migrate - outright, since a scoped run may not make the declaration and - content migrated without it cannot be decoded at all. + says so at the end when it did not, and refuses a --path/--limit + --yes migrate outright, since a scoped run may not make the + declaration and content migrated without it cannot be decoded + at all. Phase 1 feeds phase 3 for free: a row retyped to `image/gif` inherits that type's `compression = 'none'`, so phase 3's `compression = 'brotli'` selection simply @@ -171,14 +171,16 @@ def _brotli(args: list[str], payload: bytes) -> tuple[bool, bytes, str]: return done.returncode == 0, done.stdout, done.stderr.decode("utf-8", "replace").strip() -def decode_plain(payload: bytes) -> tuple[bool, bytes]: - ok, out, _ = _brotli(["-d", "-c"], payload) - return ok, out +# Both decoders hand back stderr, as encode_with_dictionary always has. Discarding it meant every +# non-zero exit read as "this row does not decode" -- so a vanished dictionary tempfile, an +# OOM-killed child or a missing brotli binary were all reported to the operator as data corruption +# in the content, which is the one explanation that sends them looking in the wrong place. +def decode_plain(payload: bytes) -> tuple[bool, bytes, str]: + return _brotli(["-d", "-c"], payload) -def decode_with_dictionary(payload: bytes) -> tuple[bool, bytes]: - ok, out, _ = _brotli(["-d", "-D", _DICTIONARY_PATH, "-c"], payload) - return ok, out +def decode_with_dictionary(payload: bytes) -> tuple[bool, bytes, str]: + return _brotli(["-d", "-D", _DICTIONARY_PATH, "-c"], payload) def encode_with_dictionary(payload: bytes, quality: int, window: int) -> tuple[bool, bytes, str]: @@ -270,7 +272,7 @@ class Result: """Phase 3's verdict on one item.""" base_path: str - status: str # migrated | already | unchanged | error + status: str # migrated | already | error slices: list[bytes] = field(default_factory=list) before: int = 0 after: int = 0 @@ -284,12 +286,13 @@ def inspect_item(item: Item, blobs: list[bytes]) -> Inspection: before = len(stored) if item.compression == "brotli": - ok, payload = decode_plain(stored) + ok, payload, plain_err = decode_plain(stored) if not ok: - ok, payload = decode_with_dictionary(stored) + ok, payload, dict_err = decode_with_dictionary(stored) if not ok: return Inspection(item.base_path, "error", before=before, - detail="decodes neither plainly nor with the dictionary") + detail=f"decodes neither plainly ({plain_err or 'no stderr'}) nor " + f"with the dictionary ({dict_err or 'no stderr'})") else: payload = stored @@ -302,7 +305,7 @@ def inspect_item(item: Item, blobs: list[bytes]) -> Inspection: before=before, after=len(payload)) -def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only_if_smaller: bool, +def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, db_was_migrated: bool) -> Result: """Decode an item, recompress it against the dictionary, and re-split it. @@ -315,13 +318,23 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only stored = b"".join(blobs) before = len(stored) - ok, plaintext = decode_plain(stored) - ok_dict, as_dict = decode_with_dictionary(stored) + ok, plaintext, plain_err = decode_plain(stored) + ok_dict, as_dict, dict_err = decode_with_dictionary(stored) + + # The mislabel census is computed on whichever decode is authoritative, and on every path that + # returns. Leaving it unset on the "already" branches meant a re-run over an already-migrated + # database reported zero mislabelled rows -- not because there were none, but because the only + # branch that filled it in was the one a migrated row never takes. + def census(content: bytes) -> str: + return sniff(content) if is_text_type(item.content_type) else "" if not ok: if ok_dict: - return Result(item.base_path, "already", before=before, after=before) - return Result(item.base_path, "error", before=before, detail="decodes neither plainly nor with the dictionary") + return Result(item.base_path, "already", before=before, after=before, + sniffed=census(as_dict)) + return Result(item.base_path, "error", before=before, + detail=f"decodes neither plainly ({plain_err or 'no stderr'}) nor with the " + f"dictionary ({dict_err or 'no stderr'})") # Both decodes succeeded and returned different bytes, neither erroring. This is not exotic: # a short stored stream (92 B of highly repetitive HTML, say) decodes plainly to the real @@ -334,24 +347,26 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only # happily validates, because it round-trips the garbage. if ok_dict and as_dict != plaintext: if db_was_migrated: - return Result(item.base_path, "already", before=before, after=before) + return Result(item.base_path, "already", before=before, after=before, + sniffed=census(as_dict)) # Decoding every row here anyway makes a mislabel sweep free: report a # text-typed row whose payload is recognisably binary, whatever its name. - mislabelled = sniff(plaintext) if is_text_type(item.content_type) else "" + mislabelled = census(plaintext) ok, recompressed, stderr = encode_with_dictionary(plaintext, quality, window) if not ok: return Result(item.base_path, "error", before=before, detail=f"compression failed: {stderr}") # The migration is only worth anything if it round-trips exactly. - ok, roundtrip = decode_with_dictionary(recompressed) + ok, roundtrip, roundtrip_err = decode_with_dictionary(recompressed) if not ok or roundtrip != plaintext: return Result( item.base_path, "error", before=before, - detail="recompressed bytes do not decode back to the original content", + detail="recompressed bytes do not decode back to the original content" + + (f" ({roundtrip_err})" if roundtrip_err else ""), ) # "Nothing to do" is a property of the RE-ENCODED stream, not the stored one. The test this @@ -364,10 +379,6 @@ def migrate_item(item: Item, blobs: list[bytes], quality: int, window: int, only if recompressed == stored: return Result(item.base_path, "already", before=before, after=before, sniffed=mislabelled) - if only_if_smaller and len(recompressed) >= before: - return Result(item.base_path, "unchanged", before=before, after=before, sniffed=mislabelled, - detail=f"dictionary-compressed form is larger ({len(recompressed)} vs {before})") - return Result(item.base_path, "migrated", slices=slice_stream(recompressed), before=before, after=len(recompressed), sniffed=mislabelled) @@ -378,7 +389,10 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: SELECT C.id, C.path, C.languageID, C.contentTypeID, C.templateId, -- IFNULL: a row with NULL content has NULL length, which made the byte totals -- (and so the phase summary) throw before read_blobs could report the row. - IFNULL(LENGTH(C.content), 0), CT.value, CT.compression + -- CAST to BLOB first: LENGTH() on a TEXT-class value counts CHARACTERS, so a row + -- holding multi-byte UTF-8 measured short, and this number decides what counts as a + -- 1 MiB chunk base. The app measures the same rows with getBlob().size, i.e. bytes. + IFNULL(LENGTH(CAST(C.content AS BLOB)), 0), CT.value, CT.compression FROM Content C JOIN ContentTypes CT ON CT.id = C.contentTypeID WHERE {predicate} @@ -391,7 +405,7 @@ def load_items(connection: sqlite3.Connection, predicate: str) -> list[Item]: # "-1" was filtered out reported first_suffix 2, so write_item re-created the stream starting # at -2 -- re-introducing the ADFA-5171 defect phase 2 had just normalised. all_rows = connection.execute( - "SELECT id, path, languageID, contentTypeID, IFNULL(LENGTH(content), 0) FROM Content" + "SELECT id, path, languageID, contentTypeID, IFNULL(LENGTH(CAST(content AS BLOB)), 0) FROM Content" ).fetchall() base_bytes_by_path = {path: length for _, path, _, _, length in all_rows} selected_ids = {row[0] for row in rows} @@ -537,7 +551,12 @@ def read_blobs(connection: sqlite3.Connection, item: Item) -> list[bytes] | None """ ids = [item.base_id] + [row_id for row_id, _, _ in item.continuations] placeholders = ",".join("?" * len(ids)) - found = dict(connection.execute(f"SELECT id, content FROM Content WHERE id IN ({placeholders})", ids).fetchall()) + # CAST to BLOB so a TEXT-class row arrives as bytes: sqlite3 hands back `str` otherwise, which + # fails in the worker at b"".join(blobs). Same reason as the LENGTH() casts in load_items -- the + # app reads these rows with getBlob(), and so should this. + found = dict(connection.execute( + f"SELECT id, CAST(content AS BLOB) FROM Content WHERE id IN ({placeholders})", ids + ).fetchall()) blobs = [found.get(row_id) for row_id in ids] return None if any(blob is None for blob in blobs) else blobs @@ -631,16 +650,6 @@ def write_item( return inserted, deleted -def retype_rows(connection: sqlite3.Connection, item: Item, content_type_id: int) -> None: - """Point an item's rows at a different content type, leaving the bytes alone.""" - ids = [item.base_id] + [row_id for row_id, _, _ in item.continuations] - placeholders = ",".join("?" * len(ids)) - connection.execute( - f"UPDATE Content SET contentTypeID = ? WHERE id IN ({placeholders})", - [content_type_id, *ids], - ) - - # The MAJOR the app requires before it will attach the dictionary at all # (DatabaseVersionResolver.MAJOR_VERSION_WITH_COMPRESSION_DICTIONARY). Migrating content without # declaring this leaves a database whose every brotli row fails to decode: WebServer gates on the @@ -908,7 +917,7 @@ def phase_retype(connection, pool, args, write, items) -> tuple[set[str], list[s return kept, errors, notes -def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> tuple[int, list[str], list[str]]: +def phase_renumber(connection, args, write, select) -> tuple[int, list[str], list[str]]: """Shift -2-based continuation numbering down to the -1 the app expects. [select] applies --limit and --path here as it does to the other phases. Without @@ -916,8 +925,6 @@ def phase_renumber(connection, args, write, retyped_paths: set[str], select) -> rewrote every chunked item in the database. """ items = select([item for item in load_items(connection, "1 = 1") if item.continuations]) - if args.renumber_scope == "retyped": - items = [item for item in items if item.base_path in retyped_paths] # load_items only groups a "-" sibling under a base holding exactly CHUNK_BYTES -- the # app's own chunk-detection rule -- so every item here is genuinely chunked, and a # coincidentally named independent page (e.g. k/kotlin-1-2 next to the real page k/kotlin-1) @@ -1022,8 +1029,6 @@ def main() -> int: parser.add_argument("--mov-type", choices=("quicktime", "mp4"), default="quicktime", help="what to call the ftypqt .mov payloads: the honest video/quicktime (inserted into " "ContentTypes) or the video/mp4 Chromium is likelier to play (default: quicktime)") - parser.add_argument("--renumber-scope", choices=("all", "retyped"), default="all", - help="renumber every -2-based chunked item, or only the ones phase 1 retyped (default: all)") parser.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 2)), help="parallel compressors") parser.add_argument("--quality", type=int, default=11, help="brotli quality (default 11, as the pipeline uses)") parser.add_argument("--window", type=int, default=22, @@ -1031,12 +1036,6 @@ def main() -> int: parser.add_argument("--limit", type=int, default=0, help="stop after this many items (for a smoke test)") parser.add_argument("--path", default="", help="only items whose base path contains this substring") parser.add_argument("--batch", type=int, default=200, help="items per write transaction") - parser.add_argument( - "--only-if-smaller", - action="store_true", - help="leave a row alone when its dictionary-compressed form is not smaller " - "(refused when the migrate phase runs: see the error it prints)", - ) args = parser.parse_args() write = args.yes and not args.dry_run @@ -1054,27 +1053,13 @@ def main() -> int: print(f"error: --phases selected no phases; pick from {', '.join(ALL_PHASES)}", file=sys.stderr) return 2 - # A migrate run declares version DICTIONARY_MAJOR_VERSION, and a plain-brotli row left behind - # in a database declaring that version can decode against the dictionary to different bytes - # *without erroring* -- served as silently wrong content. --only-if-smaller deliberately - # leaves such rows, so the two cannot travel together. - if args.only_if_smaller and "migrate" in args.phase_list: - print( - "error: --only-if-smaller cannot be combined with the migrate phase: it deliberately " - "leaves rows plain-compressed in a database the run declares version " - f"{DICTIONARY_MAJOR_VERSION}, and a plain row in such a database can decode against " - "the dictionary to wrong bytes without erroring", - file=sys.stderr, - ) - return 2 - # A run that may not declare the version may not write migrated content either -- the two # travel together or not at all. --path/--limit scope the run, so may_declare_version withholds # the declaration; phase 3 nevertheless recompressed those rows and COMMITTED them, printed a # WARNING on stdout and exited 0. Because the app gates the dictionary on the declared version # and not on the dictionary's presence, every row just rewritten decodes as "corrupt input", - # and the plaintext it was rewritten from is gone. Refused here, like --only-if-smaller above, - # rather than warned about after the damage. A scoped dry run is still useful and still allowed. + # and the plaintext it was rewritten from is gone. Refused here rather than warned about after + # the damage. A scoped dry run is still useful and still allowed. withheld = may_declare_version(args) if write and "migrate" in args.phase_list and withheld: print( @@ -1178,7 +1163,7 @@ def select(items: list[Item]) -> list[Item]: # Phase 3's tallies live out here, not next to its loop, because the summary below has to be # printable whether or not that loop ran or finished -- see the abort handling after the try. - counts = {"migrated": 0, "already": 0, "unchanged": 0, "error": 0} + counts = {"migrated": 0, "already": 0, "error": 0} wrote_migrated_content = False version_declared = False migrate_started = False @@ -1212,7 +1197,7 @@ def select(items: list[Item]) -> list[Item]: print() if "renumber" in args.phase_list: - _, phase_errors, phase_notes = phase_renumber(connection, args, write, retyped_paths, select) + _, phase_errors, phase_notes = phase_renumber(connection, args, write, select) errors += phase_errors notes += phase_notes print() @@ -1257,8 +1242,7 @@ def select(items: list[Item]) -> list[Item]: continue pending[ pool.submit( - migrate_item, item, blobs, args.quality, args.window, args.only_if_smaller, - db_was_migrated, + migrate_item, item, blobs, args.quality, args.window, db_was_migrated, ) ] = item @@ -1361,8 +1345,6 @@ def select(items: list[Item]) -> list[Item]: print(" The migrate phase had not started.") print(f"migrated {counts['migrated']:,}") print(f"already {counts['already']:,}") - if counts["unchanged"]: - print(f"left alone {counts['unchanged']:,} (not smaller with the dictionary)") print(f"errors {counts['error']:,}") if write: print(f"rows inserted {inserted_total} rows deleted {deleted_total}") @@ -1417,7 +1399,7 @@ def select(items: list[Item]) -> list[Item]: # dictionary version. attempted = sum(counts.values()) unattempted = max(0, len(items) - attempted) if migrate_started else 0 - remaining = counts["unchanged"] + len(failed_items) + unattempted + remaining = len(failed_items) + unattempted if version_declared and remaining: print( f"\nWARNING: this database declares version {DICTIONARY_MAJOR_VERSION}.x but " From 7d8d47360c3d67eaaf9b11f9e0b9a648af7a6594 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 1 Sep 2026 00:45:51 +0000 Subject: [PATCH 16/16] ADFA-5153: Replace the version row instead of appending one ADFA-5220 (#1729, merged) redefined DocumentationDatabaseVersion as an exactly-one-row table: writers replace the row, and the app warns over a file holding several. The script still appended, citing the old append-only contract, so a migrated database tripped that warning -- and a pre-existing row with a newer changeTime would outrank the new one, leaving dictionary decoding off for content that needs it. declare_dictionary_version now DELETEs the table before its INSERT, and declared_major orders by changeTime DESC, rowid DESC -- the same query DatabaseVersionResolver.resolveMajorVersion runs -- so both readers give the same answer over the same file. Both docstrings now describe the one-row contract. Verified against a scratch documentation.db-shaped fixture carrying a pre-existing major-1 row: after a full run exactly one row remains (the new 2.0.0), a second run is idempotent (still one row, 0 re-migrated), and a future-dated stray row is replaced rather than outranking the declaration. --- .../migrate_content_to_dictionary_brotli.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/scripts/docdb/migrate_content_to_dictionary_brotli.py b/scripts/docdb/migrate_content_to_dictionary_brotli.py index f1ff597edd..d66897e11d 100755 --- a/scripts/docdb/migrate_content_to_dictionary_brotli.py +++ b/scripts/docdb/migrate_content_to_dictionary_brotli.py @@ -672,14 +672,15 @@ def write_item( def declared_major(connection: sqlite3.Connection) -> int | None: """The MAJOR this database declares, or None when it declares none. - Reads the row with the highest rowid: the table is append-only by contract - (docs/documentation-database.md), so the row inserted last is the current version -- the same - row the app's DatabaseVersionResolver reads (ADFA-5220). + The table holds exactly one row by contract (ADFA-5220 / #1729, + docs/documentation-database.md). The ORDER BY is the defence for a file that breaks it, + matching the app's DatabaseVersionResolver: the newest changeTime wins, rowid breaking ties, + so both readers give the same answer over the same file. """ if not table_exists(connection, "DocumentationDatabaseVersion"): return None row = connection.execute( - "SELECT major FROM DocumentationDatabaseVersion ORDER BY rowid DESC LIMIT 1" + "SELECT major FROM DocumentationDatabaseVersion ORDER BY changeTime DESC, rowid DESC LIMIT 1" ).fetchone() return row[0] if row is not None and row[0] is not None else None @@ -706,11 +707,13 @@ def declare_dictionary_version(connection: sqlite3.Connection) -> None: Written in the same transaction as the first batch of migrated content, because the two facts have to travel together: content compressed against the dictionary, and a version saying so. - The log is append-only by contract (docs/documentation-database.md, DatabaseVersionResolver): - each change is another INSERT and the row inserted last is the current version, so prior - version rows are history to keep, never state to replace. + The table holds exactly one row by contract (ADFA-5220 / #1729, + docs/documentation-database.md, DatabaseVersionResolver): the version is the state the file + *is*, so the DELETE replaces whatever row is there rather than appending history -- the app + warns over a multi-row file. """ connection.execute(VERSION_TABLE_SQL) + connection.execute("DELETE FROM DocumentationDatabaseVersion") connection.execute( "INSERT INTO DocumentationDatabaseVersion (major, minor, patch, who, comment) VALUES (?, 0, 0, ?, ?)", (