[flink] Support negated predicates in PredicateConverter - #9427
[flink] Support negated predicates in PredicateConverter#9427Stephen0421 wants to merge 1 commit into
Conversation
| return negated | ||
| ? visitBiFunction(children, builder::equal, builder::equal) | ||
| : visitBiFunction(children, builder::notEqual, builder::notEqual); | ||
| } else if (func == BuiltInFunctionDefinitions.GREATER_THAN) { |
There was a problem hiding this comment.
[P1] Preserve Flink NaN semantics for negated comparisons
For FLOAT and DOUBLE, this rewrite is not equivalent to the original expression. Flink numeric comparisons use Java operators, so NOT (NaN > 1.0) evaluates to true, while the Paimon LessOrEqual predicate orders values through Double.compare and rejects NaN. Because this predicate is pushed down before the remaining Flink filter runs, the row is discarded and the query returns incomplete results.
Please keep negated floating-point comparisons unsupported/residual, or construct NaN-aware equivalents for every comparison direction and add row/source tests containing NaN.
| } | ||
| return builder.in(builder.indexOf(fieldRefExpr.getName()), literals); | ||
| return negated | ||
| ? builder.notIn(field.index, literals) |
There was a problem hiding this comment.
[P2] Short-circuit NOT IN lists containing NULL
Under SQL WHERE semantics, v NOT IN (1, NULL, 3) can never be true. Passing the NULL literal to builder.notIn is also unsafe for file-index evaluation: the BSI reader can unbox a null mapped value, and the range-bitmap reader can pass null to its comparator, causing the query to fail when either index is enabled. Before this change, the unsupported expression stayed as a Flink residual filter.
Please return an always-false predicate when a negated IN list contains NULL, or reject the conversion so it remains residual. An index-enabled regression test would cover both paths.
2 participants
Purpose
This PR extends
PredicateConverterto support negated predicates while preserving SQL three-valued logic.The main changes include:
NOTconversion with double-negation elimination.ANDandORexpressions.NOT INthroughPredicateBuilder.notIn.NOT BETWEENthrough the structured negation ofBetween.IS NULLandIS NOT NULL.IS NOT TRUE,IS NOT FALSE, and their unary negations with correct NULL semantics.NOT LIKE, as Flink residual filters.The conversion failure contract remains unchanged: expressions that cannot be converted safely are not consumed by the source and remain for Flink evaluation.
Tests
Added unit tests for:
INandNOT IN, including NULL literals and large-IN predicates.NOT BETWEENpredicate structure.AND/ORnegation and doubleNOT.IS TRUE,IS FALSE,IS NOT TRUE, andIS NOT FALSEover TRUE, FALSE, and NULL.NOT LIKEconversion.Added a nullable SQL integration test covering
NOT BETWEEN,NOT IN, and boolean truth predicates.Verified with Flink 1 and Flink 2:
PredicateConverterTestandFlinkTableSourceTest: 65 tests passed for each profile.ReadWriteTableITCase#testNullablePredicateThreeValuedLogic: passed for each profile.