Skip to content

FiboaDuckDBBaseConverter: convert Parquet sources with SQL - #272

Closed
ivorbosloper wants to merge 0 commit into
split/per-filefrom
split/duckdb
Closed

ivorbosloper wants to merge 0 commit into
split/per-filefrom
split/duckdb

Conversation

@ivorbosloper

@ivorbosloper ivorbosloper commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

This module is meant to be short-lived — read the last paragraph before reviewing the code.

A source that is already Parquet does not need to become a GeoDataFrame. This converter maps columns, filters rows and migrates values in SQL, writes with DuckDB, then repairs what DuckDB does not write itself: the GeoParquet and collection metadata, the nullability the schema asks for, the bbox covering column, and the canonical Hilbert order. Japan's Fude polygons are 33 GB of Parquet across four editions, which is what it was written for.

The same converter is moving into vecorel-cli in vecorel/cli#29, which already carries everything this branch learned: the Hilbert fallback for a CRS with no area_of_use, the large_binary widening for take() over 2 GB of WKB, the metadata-preserving batch write, the bbox covering declaration, closing readers before replacing a file on Windows, and the schema prewarm.

So: if #29 is merged and released before this is reviewed, close this PR#273 then needs one import line changed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG

@ivorbosloper

Copy link
Copy Markdown
Collaborator Author

vecorel/cli#29 is merged and released: vecorel-cli 0.2.18 ships vecorel_cli/conversion/duckdb.py. As this PR's own last paragraph said, that makes this copy redundant — closing it. #280 bumps the pin and points jp at the upstream class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant