[AURON #2474] Fix make_date null and ANSI semantics

[Sigma-Ma]

Which issue does this PR close?

Closes #2474

Rationale for this change

Auron currently maps Spark's MakeDate expression directly to DataFusion's stock make_date implementation. That implementation does not preserve Spark's typed or columnar NULL semantics and returns execution errors for invalid inputs in non-ANSI mode.

The current mapping also drops MakeDate.failOnError, so the native path cannot distinguish ANSI from non-ANSI behavior.

What changes are included in this PR?

• Route Spark MakeDate through a Spark_MakeDate extension function.
• Pass failOnError through the existing extension-function arguments.
• Propagate scalar and columnar NULLs for year, month, and day.
• Return NULL for invalid dates in non-ANSI mode and raise an execution error in ANSI mode.
• Add Rust and Spark regression coverage for NULL and invalid inputs.

Are there any user-facing changes?

Bug fix only. make_date now follows Spark's NULL and invalid-date behavior.
There are no public API or configuration changes.

How was this patch tested?

• cargo test -p datafusion-ext-functions --lib.
• Spark 3.5.8 / Scala 2.12 compile and test-compile passed.

./build/mvn \
  -pl spark-extension,spark-extension-shims-spark \
  -am compile test-compile \
  -DskipBuildNative \
  -Ppre -Pscala-2.12 -Pspark-3.5

• AuronFunctionSuite passed with the native engine.
• ./dev/reformat --check.
• cargo fmt --check
• cargo test -p datafusion-ext-functions test_spark_make_date --lib
• cargo test -p datafusion-ext-functions --lib

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: OpenAI Codex (GPT-5)

ASF guidance: https://www.apache.org/legal/generative-tooling.html

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Fixes Spark make_date native execution semantics by routing MakeDate through a dedicated extension function that preserves Spark NULL propagation and ANSI/non-ANSI invalid-date behavior.

Changes:

• Add a Spark shim hook to retrieve MakeDate.failOnError, and pass it through to the native engine.
• Introduce Spark_MakeDate ext function in Rust to implement Spark’s NULL + invalid-date semantics.
• Add Spark and Rust regression tests for NULL propagation and invalid inputs (ANSI vs non-ANSI).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
spark-extension/src/main/scala/org/apache/spark/sql/auron/Shims.scala	Adds shim API to extract MakeDate.failOnError.
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala	Routes MakeDate to Spark_MakeDate and forwards failOnError.
spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/auron/ShimsImpl.scala	Implements getMakeDateFailOnError across Spark versions.
native-engine/datafusion-ext-functions/src/spark_dates.rs	Implements spark_make_date with Spark-compatible NULL/ANSI behavior + tests.
native-engine/datafusion-ext-functions/src/lib.rs	Registers the new Spark_MakeDate extension function.
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronFunctionSuite.scala	Adds Spark-side regression coverage for NULL/invalid-date behavior in native mode.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Sigma-Ma]

Thanks for catching this. I replaced the Int32 downcast expect calls with DataFusionError::Execution, so an unexpected input type no longer panics the executor. I also added a regression test for this case.

[Sigma-Ma]

I replaced the unchecked as i32 conversion with NaiveDate::to_epoch_days(), which returns the Date32-compatible epoch-day value directly.

[lyne7-sc]

Thanks for the fix. The implementation looks good overall to me. I left two comments for your consideration: one about the spark 3.0-specific test expectation and one non-blocking performance suggestion.

[lyne7-sc]

should we build the result directly with a Date32Builder here?

the current Vec followed by Date32Array::from(result) requires a temporary allocation and a second full pass to pack the values and validity bitmap.

appending values and nulls directly to a Date32Builder in the existing loop would preserve the current behavior while avoiding the temporary vector and extra traversal.

[Sigma-Ma]

Thanks for the suggestion. I replaced the temporary Vec<Option<i32>> with Date32Builder and now append values and nulls directly in the existing loop.

This removes the temporary allocation and extra conversion pass without changing the scalar or array behavior.

[lyne7-sc]

This expectation does not hold on spark 3.0. MakeDate has no failOnError in that version, and the 3.0 shim intentionally passes false, so this query returns null even when ANSI mode is enabled. should we make this test version-specific, expecting null on spark 3.0 and an exception on spark 3.1+?

[Sigma-Ma]

You're right. Spark 3.0 does not expose MakeDate.failOnError, and the 3.0 shim intentionally forwards false. I updated the test to branch on AuronTestUtils.isSparkV31OrGreater: Spark 3.0 compares the native result with Spark and expects NULL, while Spark 3.1+ continues to verify the ANSI exception and the native execution plan.

[slfan1989]

Thanks for addressing the review feedback. The implementation now correctly preserves NULL propagation and ANSI/non-ANSI behavior, handles unexpected input types without panicking, and accounts for Spark 3.0’s version-specific semantics. The code looks good to me. The latest workflows are still awaiting approval, so please make sure the full CI passes before merging. LGTM otherwise.

[slfan1989]

@Sigma-Ma Thanks for the contribution! Merged into master. Welcome to the Auron community!