[server] Restore KV minRetainOffset on follower restart - #4060
Conversation
4f05624 to
162c53c
Compare
| // Followers don't initialize a KV tablet, so restore the log retention state from the | ||
| // latest committed KV snapshot. The ZooKeeper read is scheduled asynchronously because | ||
| // this method runs while holding the replica state-change lock. | ||
| restoreFollowerMinRetainOffsets(kvFollowers); |
There was a problem hiding this comment.
Could we avoid this separate ZooKeeper-backed recovery state machine by piggybacking the committed minRetainOffset on follower FetchLogResponses? The leader advances this value only after a KV snapshot is committed, and follower fetches already retry continuously (including empty long-poll responses), so every successful fetch can idempotently apply updateMinRetainOffset. The existing notification can remain the fast path, while FetchLog becomes the anti-entropy and restart-recovery path.
This follows Kafka's pattern of propagating a logical cleanup boundary by logStartOffset in FetchResponse. Fluss should use a separate optional min_retain_offset field rather than overload log_start_offset: the former is snapshot-derived deletion permission. This would remove the complex logic in this PR: pending map, ZooKeeper batch reads, scheduler/backoff, and replica-identity fencing while preserving eventual local cleanup progress.
162c53c to
3ec3085
Compare
Piggyback the committed KV snapshot retention boundary on follower FetchLog responses so restarted followers converge even when the table is idle. Keep the existing snapshot notification as the fast path. Fixes apache#4054 Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 69/69 AI-Contributed/UT: 135/135
3ec3085 to
b7e06be
Compare
Make minRetainOffset updates atomic and monotonic when fetch responses race with snapshot notifications. Exercise recovery through a real delayed empty fetch response and verify local segment cleanup. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 23/23 AI-Contributed/UT: 143/143
Summary
minRetainOffsetfrom the latest committed KV snapshot when KV followers are handled inmakeFollowers.Test Plan
ZooKeeperClientTest: 23 tests passed.ReplicaManagerTest: 41 tests passed.mvn -o -pl fluss-server -DskipTests -Dfast verifypassed (Checkstyle 0 violations and Spotless passed).🤖 AI-assisted changes - reviewed by human developer