hoglake sink: write the flush identity into the snapshot message - #146
Merged
Merged
Conversation
A hoglake snapshot from millpond carried an author and an empty message. The commit now records what the flush was: records, files, partition tuples, Arrow bytes, the flush trigger, the millpond version, the table incarnation, and the Kafka offset range of every partition in the flush. The offset ranges are the only record of which Kafka position a snapshot contains. The message is formatted before the upload, so it can never orphan an object, and it is frozen into the prepared payload, so a replay re-sends the same text.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hoglake snapshots from millpond carried
author(millpond/<table>/<ordinal>) and an emptymessage. The sink already receives the flush identity,kafka_offsetsas(topic, partition, first, last), and used it only to seed the idempotency key. Nothing recorded which Kafka offsets a snapshot contains, which is what an operator needs to answer "is offset X in the lake", audit a replay after a crash, or reconcile a gap.The message
partitionsis the number of partition tuples the flush touched (0 on an unpartitioned table; equal tofilesunder the one-file-per-tuple fanout).arrow_bytesis measured after consolidation, so it is approximately the flush gate value.triggerissize,interval,final(shutdown) orunknown.tableis the table incarnation, so a replayed range after a drop and recreate is distinguishable from the dead incarnation's snapshot.... (+k more).prepare_append_files, so a formatter error cannot orphan an uploaded object, and frozen into the prepared payload, so a retry under the same idempotency key re-sends identical text.DuckLake fleets
No behaviour change on the write path:
_sink_write_kwargspasses nothing to the DuckLake sink. One metric label is new on every destination: the shutdown flush now reportsbatches_flushed_total{trigger="final"}instead of counting undertrigger="time". The repo dashboard aggregatesby (trigger); no alert selects on the label.Tests
1270 unit (+43), 65 non-hoglake integration, 45 hoglake integration (the message is read back from
GET /v1/catalogs/{c}/snapshots, including the live table UUID and one snapshot with the same message after a lost commit response). Two review rounds; mutations on the partition count, range ordering, capture point ofarrow_bytes, the truncation guard and the trigger rename are each red.Prod-us shape: 32 ranges per pod, 665 bytes per message.