Add uuid to the typedColumns vocabulary - #145
Merged
Merged
Conversation
uuid maps to pa.uuid() with a deliberately narrow parser: canonical,
32 bare hex, {}-wrapped, or urn:uuid:-prefixed; anything else nulls with
columns_coerce_failed_total. DuckLake gets a native UUID column; the
hoglake sink still writes the bare binary(16) form until millpond pins
pyhoglake 1.3.0.
Neither sink issues DDL for the pin, so the live column type wins in
both directions. hoglake renders a uuid batch back to canonical text
against a string column and parses a string batch against a uuid
column; DuckLake parses a text batch against a UUID column before the
INSERT, because its own VARCHAR->UUID cast fails the whole insert on one
unparseable value. Sort and both filters compare on the storage type.
Arrow errors are non-retryable.
pyhoglake 1.3.0 returns pa.uuid() for a uuid column and accepts both the annotated and bare spellings on append, so the batch's extension column passes through _prepare unchanged and the uploaded object carries the UUID logical type. The canary test flips to the positive assertion and the pyarrow floor moves to 21 with it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
typedColumnsgainsuuid. Until now the eventsuuidandperson_idcolumns reached both destinations as text; PostHog/charts#16185 pins them touuidon the dev hoglake instance and this is the millpond half. Two commits: the coercer and both sinks, then the pyhoglake 1.3.0 pin that makes the hoglake objects carry the parquet UUID annotation (PostHog/hoglake#185).Coercion
uuidmaps topa.uuid()(storagefixed_size_binary(16)).arrow_converter.uuid_bytesaccepts exactly four shapes: canonical 8-4-4-4-12, 32 bare hex, either wrapped in{}, or prefixed with case-sensitiveurn:uuid:. It is deliberately narrower thanuuid.UUID, which stripsurn:/uuid:and hyphens from anywhere and so remaps non-UUIDs (0x…,1_2345…, misplaced hyphens) to a UUID nobody sent; those go to NULL withcolumns_coerce_failed_total. Sources: string, large_string, binary, large_binary (wrong widths null per value). Variant and version bits are not validated; ClickHouseUUIDis an opaque 128-bit value.Per uuid column on 27k rows: 11.8 ms warm, 0.44 µs/value.
Destinations
Neither sink issues DDL for the pin, so the live column type wins in both directions.
arrow.uuidbecomes a nativeUUIDcolumn on a new table. Auuidbatch against a live VARCHAR column lands as canonical text (DuckDB casts UUID to VARCHAR on insert) at the cost of one refused ALTER per flush, counted onerrors_total{type="schema"}. A text batch against a live UUID column (rollback of the pin, or an unpinned pod on a mixed fleet) is parsed by the same coercer before the INSERT: DuckDB's own VARCHAR to UUID cast fails the whole insert on one unparseable value, deterministically on every retry, and refuses theurn:uuid:form.uuidbatch against a livestringcolumn is rendered back to canonical text; astringbatch against a liveuuidcolumn is parsed withuuid_bytes, nulling unparseables as the coercer would. Each direction warns once per column and bumpserrors_total{type="schema"}per flush. Only an already-canonical lowercase hyphenated string survives a round trip; uppercase, bare-32, braces and urn all normalize. Getting a realuuidcolumn needs the table created with the pin. The uploaded parquet carries theUUIDlogical type: this pinspyhoglake>=1.3.0(and itspyarrow>=21floor, moving the uvexclude-newer-packagecarve-out past 1.3.0's publish time), so_prepare's cast to the catalog schema passes the extension column through unchanged. A barebinary(16)file from an older writer still prepares against the same table (integration-tested both ways).is_retryablenamesArrowInvalid,ArrowTypeErrorandArrowNotImplementedErroras permanent; the last fell through to the transient default before.Sort and filters
MILLPOND_SORT_BYon a uuid column sorts on storage (bytewise big-endian equals canonical text order); an unsortable key type skips withsort_skipped_total{reason="unsortable_type"}instead of killing the consume loop. Keep and drop filters on a uuid-pinned field compare on storage. An unparseable configured or polled value is dropped from the comparison and the rest of the set still applies, counted onerrors_total{type="filter_value_invalid"}with a bounded warn-once; only a set with nothing usable falls back to the filter's direction (keep admits nothing underrecords_skipped_total{reason="filter_value_invalid"}, drop denies nothing). The parsed value array is memoized per values tuple. A static non-UUID filter value on a uuid-pinned field is a startup error; dynamic poll values fail closed at runtime.filter_matched_total{value=}renders canonical text.Unpinned path
Nothing changes on a fleet without
uuidintypedColumns: the extension branches are no-ops when no column is pinned, the DuckLake realignment returns before any work when the cached live schema has no UUID column,_finish_keep_filteris value-identical, and thefixed_size_binarytoBLOBDDL branch cannot fire because millpond never emits that type.Tests
1232 unit, 41 hoglake integration; ruff clean. Four adversarial review rounds, the last two focused on the two-way handling. Mutation-tested: each hyphen offset, the 16-byte length guard, the width guard, first-chunk-only and first-column-only rewrites, the metric label renderer, the drop-filter direction, the config check's field scoping, the warn-once dedup, and the BLOB/UUID DDL mapping are each red.