Skip to content

fix: increase AVG MathContext precision from 10 to 34 to prevent BIGINT overflow and silent precision loss - #39765

Open
waterWang wants to merge 1 commit into
apache:masterfrom
waterWang:fix/avg-precision-math-context-34
Open

fix: increase AVG MathContext precision from 10 to 34 to prevent BIGINT overflow and silent precision loss#39765
waterWang wants to merge 1 commit into
apache:masterfrom
waterWang:fix/avg-precision-math-context-34

Conversation

@waterWang

Copy link
Copy Markdown

Problem

#39751 — SQL AVG rounds every result to 10 significant digits via the fixed MathContext(10, RoundingMode.HALF_UP) at BeamBuiltinAggregations.java:93. All AVG subtypes (INT32, INT64, INT16, INT8, FLOAT, DOUBLE, DECIMAL) go through prepareOutput which divides the accumulator with this precision.

Consequences:

  • AVG over a single BIGINT value 9223372036854775807 returns -9223372036709551616sign flipped due to 10-digit rounding then .longValue() truncation
  • Values above 10 significant digits (ids, epoch-millis, cent-denominated amounts) are silently rounded (e.g. AVG(1786500000123)1786500000000)
  • DECIMAL(18,2) values lose precision (123456789.99123456790.0)

Fix

Increase the MathContext precision from 10 to 34 significant digits (MathContext.DECIMAL128 precision, with the same HALF_UP rounding mode). This keeps all values representable in any of the supported input types exact through the division:

  • INT64/BIGINT values need up to 19 significant digits — 34 is sufficient, so the sign flip and truncation disappear
  • DECIMAL(38,10) can need up to 49 digits, but 34 covers the overwhelmingly common DECIMAL(p,s) ranges and matches MathContext.DECIMAL128 semantics (sum/count where the division terminates is exact; non-terminating divisions are rounded to 34 digits instead of 10)

The alternative of dropping the MathContext would turn non-terminating divisions into ArithmeticException; threading the declared output RelDataType precision/scale from AggregateCall is a larger design change touching VAR_*, STDDEV_*, COVAR_* coders — a follow-up for the component owners. This fix addresses the reported correctness bugs with a minimal, targeted change.

Fixes #39751

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @Abacn for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: SQL AVG rounds to 10 significant digits, flipping the sign on large BIGINT values

1 participant