Implement DuckDB-based converter

[ivorbosloper]

Adds a DuckDB-based converter for datasets that are too large for the GeoDataFrame-based codepath. The dataset with the largest dataframe (Japan) no longer needs a 128GB+ machine; it converts on a laptop, bounded by DuckDB's external processing, and runs in well under 30 minutes.

The implementation has moved to vecorel-cli (DuckDBBaseConverter, GeoParquet.postprocess(), Hilbert helpers); this PR keeps only the thin fiboa subclass and the jp dataset using it. The outputs of the two codepaths are comparable by construction — same schema, data types, nullability, row order (CRS-derived Hilbert grid with stable tie-breaking), collection and GeoParquet metadata (incl. CRS), and file packaging — verified by tests/test_convert_parity.py, which converts jp through both codepaths and compares the results.

⚠️ Merge order: requires vecorel-cli 0.2.18 (vecorel/cli#29) on PyPI first. After the release, run pixi update to refresh pixi.lock — CI stays red until then.

Also drops the redundant sources from the HR converter, which vecorel-cli 0.2.18 rejects because sources silently overrides the variants.

🤖 Generated with Claude Code

[m-mohr]

You can set KV_METADATA in DuckDB? That's awesome and resolves the primary issue we had on our list! Will test later, thanks! Maybe it makes sense to jump on a call for the metadata discussion.

[ivorbosloper]

You can set KV_METADATA in DuckDB? That's awesome and resolves the primary issue we had on our list! Will test later, thanks! Maybe it makes sense to jump on a call for the metadata discussion.

Yes. I first tried the python API, but that doesn't expose the Metadata option. Would be great to have a call on the metadata details, you wrote the vecorel-parquet logic and seem well informed...

[ivorbosloper]

I'm still stuck on fiboa validation:

fiboa validate jptest.parquet     
fiboa CLI 0.20.1 - Validator

Validating jptest.parquet: INVALID
 - geometry: Nullability differs, is True but must be False
 - id: Nullability differs, is True but must be False

As I understand it, the validator takes the parquet schema and checks it with the fiboa schema (see implementation ). The fiboa schema is built from the extensions (and is correct in this case), but the parquet schema is implicitly created by duckdb (with TO 'file.parquet' (FORMAT parquet)) based on the resultset of the query.

Can I force this to a 'non-null' result? I've tried Casting with ::VARCHAR NOT NULL, but that's not the correct syntax.

Maybe related to duckdb/duckdb#13949

[ivorbosloper]

I think duckdb parquet writer doesn't support setting the nullability derived from the resultset.

$ duckdb
D CREATE TABLE test(id varchar not null);
D insert into test values ('A'), ('B');
D COPY (select * from test) to '/tmp/test.parquet' (FORMAT PARQUET);
D exit

$ parquet-dump-schema /tmp/test.parquet 
required group field_id=-1 duckdb_schema {
  optional binary field_id=-1 id (String);
}

same for COPY test to '/tmp/test.parquet' (FORMAT PARQUET)

[m-mohr]

For me that sounds like a bug in duckdb. Is there an open issue for it, otherwise maybe open one?

[ivorbosloper]

For me that sounds like a bug in duckdb. Is there an open issue for it, otherwise maybe open one?

Is "Non-nullability" a property of a query result column? Maybe this information is lost. But at least it's a feature request..

[m-mohr]

@ivorbosloper In which state is this PR? Was the collection-level metadata and column schema ever solved? Is this generating valid fiboa/vecorel files?

[m-mohr]

Updated PR to work with new duckdb implementation in vecorel-cli, needs a release of vecorel-cli first.

[ivorbosloper]

Superseded. The converter now lands in two places: FiboaDuckDBBaseConverter in #272 (rebased on #266, with the JP converter that uses it in #273), and the generic version m-mohr moved upstream in vecorel/cli#29. Once vecorel/cli#29 is released, fiboa's copy goes away and #273 imports the upstream class. Closing this one so the three of them do not compete.