Skip to content

[server] Restore KV minRetainOffset on follower restart - #4060

Open
gyang94 wants to merge 2 commits into
apache:mainfrom
gyang94:codex/issue-4054-startup-restore
Open

[server] Restore KV minRetainOffset on follower restart#4060
gyang94 wants to merge 2 commits into
apache:mainfrom
gyang94:codex/issue-4054-startup-restore

Conversation

@gyang94

@gyang94 gyang94 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Restore follower minRetainOffset from the latest committed KV snapshot when KV followers are handled in makeFollowers.
  • Coalesce pending restores into one asynchronous task per TabletServer, keeping ZooKeeper access outside the replica state-change lock.
  • Bound ZooKeeper pressure with 32-bucket client batches, the configured max-inflight limit, startup jitter, and exponential retry backoff.
  • Distinguish confirmed-empty snapshots from unresolved reads, and ignore stale results after notifications, role changes, replica replacement, or shutdown.
  • Fixes [server] Restore KV minRetainOffset on follower restart #4054

Test Plan

  • ZooKeeperClientTest: 23 tests passed.
  • ReplicaManagerTest: 41 tests passed.
  • mvn -o -pl fluss-server -DskipTests -Dfast verify passed (Checkstyle 0 violations and Spotless passed).

🤖 AI-assisted changes - reviewed by human developer

@gyang94
gyang94 marked this pull request as ready for review August 25, 2026 11:57
@gyang94
gyang94 force-pushed the codex/issue-4054-startup-restore branch from 4f05624 to 162c53c Compare August 31, 2026 04:15
// Followers don't initialize a KV tablet, so restore the log retention state from the
// latest committed KV snapshot. The ZooKeeper read is scheduled asynchronously because
// this method runs while holding the replica state-change lock.
restoreFollowerMinRetainOffsets(kvFollowers);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid this separate ZooKeeper-backed recovery state machine by piggybacking the committed minRetainOffset on follower FetchLogResponses? The leader advances this value only after a KV snapshot is committed, and follower fetches already retry continuously (including empty long-poll responses), so every successful fetch can idempotently apply updateMinRetainOffset. The existing notification can remain the fast path, while FetchLog becomes the anti-entropy and restart-recovery path.

This follows Kafka's pattern of propagating a logical cleanup boundary by logStartOffset in FetchResponse. Fluss should use a separate optional min_retain_offset field rather than overload log_start_offset: the former is snapshot-derived deletion permission. This would remove the complex logic in this PR: pending map, ZooKeeper batch reads, scheduler/backoff, and replica-identity fencing while preserving eventual local cleanup progress.

@gyang94
gyang94 force-pushed the codex/issue-4054-startup-restore branch from 162c53c to 3ec3085 Compare September 1, 2026 02:12
Piggyback the committed KV snapshot retention boundary on follower FetchLog
responses so restarted followers converge even when the table is idle. Keep
the existing snapshot notification as the fast path.

Fixes apache#4054

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-5
AI-Contributed/Feature: 69/69
AI-Contributed/UT: 135/135
@gyang94
gyang94 force-pushed the codex/issue-4054-startup-restore branch from 3ec3085 to b7e06be Compare September 1, 2026 02:15
Make minRetainOffset updates atomic and monotonic when fetch responses
race with snapshot notifications. Exercise recovery through a real delayed
empty fetch response and verify local segment cleanup.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-5
AI-Contributed/Feature: 23/23
AI-Contributed/UT: 143/143
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[server] Restore KV minRetainOffset on follower restart

2 participants