Skip to content

Add uuid to the typedColumns vocabulary - #145

Merged
jghoman merged 2 commits into
mainfrom
jakob/typed-uuid
Sep 23, 2026
Merged

jghoman merged 2 commits into
mainfrom
jakob/typed-uuid

Conversation

@jghoman

@jghoman jghoman commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

typedColumns gains uuid. Until now the events uuid and person_id columns reached both destinations as text; PostHog/charts#16185 pins them to uuid on the dev hoglake instance and this is the millpond half. Two commits: the coercer and both sinks, then the pyhoglake 1.3.0 pin that makes the hoglake objects carry the parquet UUID annotation (PostHog/hoglake#185).

Coercion

uuid maps to pa.uuid() (storage fixed_size_binary(16)). arrow_converter.uuid_bytes accepts exactly four shapes: canonical 8-4-4-4-12, 32 bare hex, either wrapped in {}, or prefixed with case-sensitive urn:uuid:. It is deliberately narrower than uuid.UUID, which strips urn:/uuid: and hyphens from anywhere and so remaps non-UUIDs (0x…, 1_2345…, misplaced hyphens) to a UUID nobody sent; those go to NULL with columns_coerce_failed_total. Sources: string, large_string, binary, large_binary (wrong widths null per value). Variant and version bits are not validated; ClickHouse UUID is an opaque 128-bit value.

Per uuid column on 27k rows: 11.8 ms warm, 0.44 µs/value.

Destinations

Neither sink issues DDL for the pin, so the live column type wins in both directions.

  • DuckLake: arrow.uuid becomes a native UUID column on a new table. A uuid batch against a live VARCHAR column lands as canonical text (DuckDB casts UUID to VARCHAR on insert) at the cost of one refused ALTER per flush, counted on errors_total{type="schema"}. A text batch against a live UUID column (rollback of the pin, or an unpinned pod on a mixed fleet) is parsed by the same coercer before the INSERT: DuckDB's own VARCHAR to UUID cast fails the whole insert on one unparseable value, deterministically on every retry, and refuses the urn:uuid: form.
  • hoglake: a uuid batch against a live string column is rendered back to canonical text; a string batch against a live uuid column is parsed with uuid_bytes, nulling unparseables as the coercer would. Each direction warns once per column and bumps errors_total{type="schema"} per flush. Only an already-canonical lowercase hyphenated string survives a round trip; uppercase, bare-32, braces and urn all normalize. Getting a real uuid column needs the table created with the pin. The uploaded parquet carries the UUID logical type: this pins pyhoglake>=1.3.0 (and its pyarrow>=21 floor, moving the uv exclude-newer-package carve-out past 1.3.0's publish time), so _prepare's cast to the catalog schema passes the extension column through unchanged. A bare binary(16) file from an older writer still prepares against the same table (integration-tested both ways).
  • is_retryable names ArrowInvalid, ArrowTypeError and ArrowNotImplementedError as permanent; the last fell through to the transient default before.

Sort and filters

MILLPOND_SORT_BY on a uuid column sorts on storage (bytewise big-endian equals canonical text order); an unsortable key type skips with sort_skipped_total{reason="unsortable_type"} instead of killing the consume loop. Keep and drop filters on a uuid-pinned field compare on storage. An unparseable configured or polled value is dropped from the comparison and the rest of the set still applies, counted on errors_total{type="filter_value_invalid"} with a bounded warn-once; only a set with nothing usable falls back to the filter's direction (keep admits nothing under records_skipped_total{reason="filter_value_invalid"}, drop denies nothing). The parsed value array is memoized per values tuple. A static non-UUID filter value on a uuid-pinned field is a startup error; dynamic poll values fail closed at runtime. filter_matched_total{value=} renders canonical text.

Unpinned path

Nothing changes on a fleet without uuid in typedColumns: the extension branches are no-ops when no column is pinned, the DuckLake realignment returns before any work when the cached live schema has no UUID column, _finish_keep_filter is value-identical, and the fixed_size_binary to BLOB DDL branch cannot fire because millpond never emits that type.

Tests

1232 unit, 41 hoglake integration; ruff clean. Four adversarial review rounds, the last two focused on the two-way handling. Mutation-tested: each hyphen offset, the 16-byte length guard, the width guard, first-chunk-only and first-column-only rewrites, the metric label renderer, the drop-filter direction, the config check's field scoping, the warn-once dedup, and the BLOB/UUID DDL mapping are each red.

uuid maps to pa.uuid() with a deliberately narrow parser: canonical,
32 bare hex, {}-wrapped, or urn:uuid:-prefixed; anything else nulls with
columns_coerce_failed_total. DuckLake gets a native UUID column; the
hoglake sink still writes the bare binary(16) form until millpond pins
pyhoglake 1.3.0.

Neither sink issues DDL for the pin, so the live column type wins in
both directions. hoglake renders a uuid batch back to canonical text
against a string column and parses a string batch against a uuid
column; DuckLake parses a text batch against a UUID column before the
INSERT, because its own VARCHAR->UUID cast fails the whole insert on one
unparseable value. Sort and both filters compare on the storage type.
Arrow errors are non-retryable.
pyhoglake 1.3.0 returns pa.uuid() for a uuid column and accepts both the
annotated and bare spellings on append, so the batch's extension column
passes through _prepare unchanged and the uploaded object carries the
UUID logical type. The canary test flips to the positive assertion and
the pyarrow floor moves to 21 with it.
@jghoman
jghoman merged commit 22cda30 into main Sep 23, 2026
17 checks passed
@jghoman
jghoman deleted the jakob/typed-uuid branch September 23, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant