Adopt vecorel-cli 0.2.18, and convert JP through its DuckDB converter

CHANGELOG.md

@@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+- Adopt vecorel-cli 0.2.18, which carries the checks this repository was growing its own copies of: rows that cannot validate are dropped bounded by `max_dropped_share`, the required properties come from the declared schemas, ids are checked for uniqueness, a converter may not declare both `sources` and `variants`, and the schemas are fetched before any source data
+- JP: convert through vecorel-cli's DuckDB converter instead of a copy of it in this repository
+- HR: drop the rolling `sources`, which overruled every `--variant`
+- Europe-LAND: use the crop name as the crop code where the release ships an empty `crop_code` (LT 2024)
 - SE: eleven editions, 2015-2025 — the campaign is a filter on one WFS layer, so every year the service holds is a variant (it answers 2015 through 2025), over https because the http URL redirects
 - BG: the ministry's GeoServer publishes Agricultural_Land_<year> for 2021-2025, not the Arable_Land_2024 the converter asked for; the 2021 and 2022 layers are a different release again (block and usage in one ELGIDENT field, with an area column the later ones lack), PHBIDENT identifies the block rather than the polygon so it is published as block_id, and the Bulgarian names need UTF-8 forced because GeoServer writes the charset into a .cst file GDAL does not read
 - 
@@ -52,6 +56,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - ES-CAT: the 2024 download is a shapefile package, and 34 crop names new in that edition are mapped
 - DE-NDS: give the collection an id (the row index), which it was published without
 - DE-BB: ref_ident holds the FLIK (field block reference), not a farmer, and the shapefile is cp1252
+- JP: editions 2021-2024, each with the determination date of the parcel rather than a constant
+- PerFileBaseConverter: convert a multi-file source one file at a time and merge the parts, so a dataset larger than memory can be converted; used by the Spain-wide converter
+- FiboaDuckDBBaseConverter: convert a source that is already Parquet with SQL, without loading it into memory
 - Update vecorel-cli to v0.2.17:
   - GeoJSON is read as UTF-8 as the format mandates, instead of the platform locale (cp1252 on Windows mangled umlauts)
   - GeoJSON files with a byte order mark no longer fail to read

fiboa_cli/datasets/commons/euro_land.py

@@ -47,3 +47,11 @@ def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         provider = "Europe-LAND HE Project <https://doi.org/10.5281/zenodo.14230620>"
         self.provider = (f"{self.provider}, {provider}") if self.provider else provider
+
+    def migrate(self, gdf):
+        # Some Europe-LAND files (e.g. LT 2024) ship an empty crop_code column next to
+        # a populated crop_name; the name is then the best available crop code.
+        if "crop_code" in gdf.columns and gdf["crop_code"].isna().all():
+            self.warning("crop_code is empty, using crop_name as crop:code")
+            gdf["crop_code"] = gdf["crop_name"]
+        return super().migrate(gdf)

fiboa_cli/datasets/hr.py

@@ -7,7 +7,6 @@
 
 
 class Converter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
-    sources = "https://www.apprrr.hr/wp-content/uploads/nipp/land_parcels.gpkg"
     variants = {
         "2024": f"{base}/land_parcels.gpkg",
         **{str(y): f"{base}/arkod_31_12_{y}.gpkg" for y in range(2023, 2010, -1)},

fiboa_cli/datasets/jp.py

@@ -1,9 +1,9 @@
-import pandas as pd
+from vecorel_cli.conversion.duckdb import DuckDBBaseConverter
 
 from ..conversion.fiboa_converter import FiboaBaseConverter
 
 
-class JPConverter(FiboaBaseConverter):
+class JPConverter(DuckDBBaseConverter, FiboaBaseConverter):
     variants = {
         "2024": "https://data.source.coop/pacificspatial/field-polygon-jp/parquet/jp_field_polygons_2024.parquet",
         "2023": "https://data.source.coop/pacificspatial/field-polygon-jp/parquet/jp_field_polygons_2023.parquet",
@@ -32,21 +32,13 @@ class JPConverter(FiboaBaseConverter):
         "local_government_cd": "admin_local_code",
         "issue_year": "determination:datetime",
     }
+    # SQL migrations (DuckDB converter): per-feature determination date from the issue year
     column_migrations = {
-        "issue_year": lambda col: pd.to_datetime(col, format="%Y"),
+        "issue_year": "make_timestamp(CAST(issue_year AS INTEGER), 1, 1, 0, 0, 0) AT TIME ZONE 'UTC'",
     }
-
     missing_schemas = {
         "properties": {
             "land_type_en": {"type": "string"},
             "admin_local_code": {"type": "string"},
         }
     }
-
-    def convert(self, *args, **kwargs):
-        # Open only these columns to limit memory usage
-        super().convert(
-            *args,
-            columns=["GEOM", "polygon_uuid", "land_type_en", "local_government_cd", "issue_year"],
-            **kwargs,
-        )