Skip to content

[server] Restore KV minRetainOffset on follower restart#4054

Description

@gyang94

Search before asking

  • I searched in the issues and found nothing similar.

Fluss version

0.9.0 (latest release)

Please describe the bug 馃悶

For a primary-key table, LogTablet.minRetainOffset is initialized to 0 after a TabletServer restart.

The leader restores the latest KV snapshot and advances this offset, but followers do not restore the snapshot offset. They only advance it through NotifyKvSnapshotOffsetRequest when a new snapshot is committed.

After a rolling upgrade with no new writes, no new KV snapshot is created and followers keep minRetainOffset = 0. TTL can roll the expired active segment on every replica, but followers cannot delete the old segment, while the leader deletes it successfully.

Reproduction:

  1. Write data to a primary-key table and wait for a KV snapshot at offset X.
  2. Stop writing data.
  3. Restart or upgrade all TabletServers.
  4. Wait for TTL to roll the expired active segment.

Expected: all replicas can delete the expired segment covered by snapshot X.

Actual: only the leader deletes it; followers retain it because their in-memory minRetainOffset remains 0.

Solution

Restore or replay the latest committed KV snapshot offset when a follower starts or joins the replica set. The offset must come from a committed KV snapshot and must not be advanced directly by log rolling. Snapshot-offset notifications should also be retried on failure.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions