Skip to content

fix: bound the usage event rewind to prevent unbounded re-aggregation - #13927

Open
waterWang wants to merge 1 commit into
apache:mainfrom
waterWang:fix-13906-usage-rewind-bound
Open

fix: bound the usage event rewind to prevent unbounded re-aggregation#13927
waterWang wants to merge 1 commit into
apache:mainfrom
waterWang:fix-13906-usage-rewind-bound

Conversation

@waterWang

Copy link
Copy Markdown

Fixes #13906

Problem

UsageManagerImpl.parse() rewinds the aggregation start date to the oldest unprocessed event, but has no bound on how far back it can rewind. If an event cannot be successfully processed, the rewind pins the window to that event's date permanently. Each subsequent run re-aggregates from that date to the present, growing by one aggregation period per run while the job keeps reporting success = 1.

Reported symptoms (4.22.1.0):

Fix

Bound the rewind to a 24-hour window. The rewind exists to absorb clock skew between the cloud and usage databases (events created during the previous run window), not to replay the full history. Events older than 24h from the current window start will still be retried, but the aggregation window will not be rewound to them.

startDateMillis = Math.max(oldestEventDate.getTime(), startDateMillis - MAX_EVENT_REWIND_MILLIS);

Testing

  • Unit test added? No (existing test class only covers event handlers; parse() has no prior tests)
  • Verified the constant is referenced exactly twice (declaration + rewind bound)
  • No behavior change for events within the 24h boundary window — they still rewind the window as before

UsageManagerImpl.parse() rewinds the aggregation start date to the
oldest unprocessed event, but has no bound on how far back it can go.
If an event cannot be successfully processed (e.g. references a removed
entity), the rewind pins the window to that event's date permanently.
Each subsequent run re-aggregates from that date to the present, growing
by one aggregation period per run. This causes unbounded growth of
cloud_usage (54M+ rows reported) and exec_time (42+ minutes per hour).

Fix: bound the rewind to 24 hours. The rewind exists to absorb clock
skew between the cloud and usage databases, not to replay history.
Events older than 24 hours from the current window start will still be
retried, but the aggregation window will not be rewound to them.

Fixes apache#13906

Signed-off-by: waterWang <waterwang@proton.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UsageManagerImpl.parse() rewind to the oldest unprocessed usage event is unbounded

1 participant