Search before asking
Fluss version
0.9.0 (latest release)
Please describe the bug 馃悶
For a primary-key table, LogTablet.minRetainOffset is initialized to 0 after a TabletServer restart.
The leader restores the latest KV snapshot and advances this offset, but followers do not restore the snapshot offset. They only advance it through NotifyKvSnapshotOffsetRequest when a new snapshot is committed.
After a rolling upgrade with no new writes, no new KV snapshot is created and followers keep minRetainOffset = 0. TTL can roll the expired active segment on every replica, but followers cannot delete the old segment, while the leader deletes it successfully.
Reproduction:
- Write data to a primary-key table and wait for a KV snapshot at offset
X.
- Stop writing data.
- Restart or upgrade all TabletServers.
- Wait for TTL to roll the expired active segment.
Expected: all replicas can delete the expired segment covered by snapshot X.
Actual: only the leader deletes it; followers retain it because their in-memory minRetainOffset remains 0.
Solution
Restore or replay the latest committed KV snapshot offset when a follower starts or joins the replica set. The offset must come from a committed KV snapshot and must not be advanced directly by log rolling. Snapshot-offset notifications should also be retried on failure.
Are you willing to submit a PR?
Search before asking
Fluss version
0.9.0 (latest release)
Please describe the bug 馃悶
For a primary-key table,
LogTablet.minRetainOffsetis initialized to0after a TabletServer restart.The leader restores the latest KV snapshot and advances this offset, but followers do not restore the snapshot offset. They only advance it through
NotifyKvSnapshotOffsetRequestwhen a new snapshot is committed.After a rolling upgrade with no new writes, no new KV snapshot is created and followers keep
minRetainOffset = 0. TTL can roll the expired active segment on every replica, but followers cannot delete the old segment, while the leader deletes it successfully.Reproduction:
X.Expected: all replicas can delete the expired segment covered by snapshot
X.Actual: only the leader deletes it; followers retain it because their in-memory
minRetainOffsetremains0.Solution
Restore or replay the latest committed KV snapshot offset when a follower starts or joins the replica set. The offset must come from a committed KV snapshot and must not be advanced directly by log rolling. Snapshot-offset notifications should also be retried on failure.
Are you willing to submit a PR?