fix: bound the usage event rewind to prevent unbounded re-aggregation - #13927
Open
waterWang wants to merge 1 commit into
Open
fix: bound the usage event rewind to prevent unbounded re-aggregation#13927waterWang wants to merge 1 commit into
waterWang wants to merge 1 commit into
Conversation
UsageManagerImpl.parse() rewinds the aggregation start date to the oldest unprocessed event, but has no bound on how far back it can go. If an event cannot be successfully processed (e.g. references a removed entity), the rewind pins the window to that event's date permanently. Each subsequent run re-aggregates from that date to the present, growing by one aggregation period per run. This causes unbounded growth of cloud_usage (54M+ rows reported) and exec_time (42+ minutes per hour). Fix: bound the rewind to 24 hours. The rewind exists to absorb clock skew between the cloud and usage databases, not to replay history. Events older than 24 hours from the current window start will still be retried, but the aggregation window will not be rewound to them. Fixes apache#13906 Signed-off-by: waterWang <waterwang@proton.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #13906
Problem
UsageManagerImpl.parse()rewinds the aggregation start date to the oldest unprocessed event, but has no bound on how far back it can rewind. If an event cannot be successfully processed, the rewind pins the window to that event's date permanently. Each subsequent run re-aggregates from that date to the present, growing by one aggregation period per run while the job keeps reportingsuccess = 1.Reported symptoms (4.22.1.0):
start_millisnever advanced past a single event 71 days earlierexec_timeup to 42.6 min per hourly run, growing by 24 aggregation periods/daycloud_usageat 54.4M rows / 14 GB, mostly duplicate replayed rowsFix
Bound the rewind to a 24-hour window. The rewind exists to absorb clock skew between the cloud and usage databases (events created during the previous run window), not to replay the full history. Events older than 24h from the current window start will still be retried, but the aggregation window will not be rewound to them.
Testing