fix: reject duplicate field names during CSV and Parquet schema inference

[sovsparrow]

Which issue does this PR close?

• Closes Parquet scan silently drops one of two columns with the same name #24381.
• Closes CSVReader behavior with dataset that has duplicate column headers is confusing #12852.

Rationale for this change

When a Parquet file has multiple columns with the same name and compatible types, DataFusion merges them during schema inference. The scan succeeds without warning but returns only one of those columns.

PyArrow can produce such a file via its public API:

left  = pa.table({"id": [1, 2, 3], "value": [10, 20, 30]})
right = pa.table({"id": [1, 2, 3], "value": [100, 200, 300]})
joined = left.join(right, keys="id")   # left_suffix/right_suffix default to None
pq.write_table(joined, path)           # writes id, value, value

DataFusion 54.0.0 returns ['id', 'value'] for this file; the [100, 200, 300] column is missing.

Other readers either preserve the data or reject the file:

Reader	Result
pyarrow ParquetFile.read()	keeps all three columns
pyarrow pq.read_table()	ArrowInvalid: Multiple matches for FieldRef.Name(value)
pyarrow dataset	ArrowInvalid: Can't unify schema with duplicate field names
duckdb	keeps all three, renames the second to value_1
polars	DuplicateError
datafusion	succeeds, missing column(s)

The same bug is behind two issues #24381 and #12852. CSV and Parquet schema inference both pass each file's schema to Schema::try_merge, which matches fields by name. Duplicate names in one file are merged before the scan. Thus, column(s) disappear silently.

What changes are included in this PR?

This PR adds ensure_unique_field_names to datafusion-datasource. It is called for each inferred CSV and Parquet schema before the merge. It uses the existing SchemaError::DuplicateUnqualifiedField error and adds the file location to the message.

Duplicate names are rejected. The PR does not rename columns or change how fields with unique names are merged across files.

Are these changes tested?

Yes. Two regression tests in order to cover the CSV and Parquet paths:

• datafusion/core/src/datasource/file_format/parquet.rs — infer_schema_rejects_duplicate_field_names
• datafusion/core/src/datasource/file_format/csv.rs — infer_schema_rejects_duplicate_header_names

Both tests create their inputs in a temporary directory; no fixtures are added. Both fail on main and pass with this change. On main, the CSV test infers Schema { fields: [id, value] } from a three-column header, matching the behavior reported in #12852.

Are there any user-facing changes?

Yes. CSV and Parquet schema inference now returns a clear error when an inferred
file schema repeats a field name. The error names the file and the repeated
column instead of returning an incomplete schema.

[Jefffrey]

this makes sense to me; i hope its a niche enough bug to not cause too much issues for downstream users now that itll error, though i suppose the other solution of renaming the duplicates would be breaking too anyway

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.84615% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.19%. Comparing base (7c079f7) to head (fda5111).
⚠️ Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/core/src/datasource/file_format/csv.rs	89.47%	0 Missing and 2 partials ⚠️
...afusion/core/src/datasource/file_format/parquet.rs	91.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24387      +/-   ##
==========================================
- Coverage   81.19%   81.19%   -0.01%     
==========================================
  Files        1110     1110              
  Lines      388739   388815      +76     
  Branches   388739   388815      +76     
==========================================
+ Hits       315636   315688      +52     
- Misses      54515    54535      +20     
- Partials    18588    18592       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sovsparrow]

Yep, that was my reasoning too. This seemed safer than renaming the duplicates, and silently dropping columns is the worst option imo. In the case of downstream issues, I’d be cool with following up. Ty.