Simplify fiboa publish for catalog-driven publication - #214
Open
ivorbosloper wants to merge 159 commits into
Open
ivorbosloper wants to merge 159 commits into
ivorbosloper wants to merge 159 commits into
Conversation
Co-authored-by: Matthias Mohr <matthias@mohr.ws>
# Conflicts: # CHANGELOG.md # pixi.lock
…easing memory requirements for large data sets
`fiboa publish` now only converts, validates, builds PMTiles and writes a collection.json with relative links, file:size/file:checksum (multihash), a web-map-links v1.3.0 `pmtiles` link with `pmtiles:layers` and a `visual` asset. README/LICENSE generation, the data-survey lookup and the S3 upload are gone; catalogs such as fieldsoftheworld/harmonized-field-data-catalog own those. spdx-license-list is only needed by tests and moves to the dev feature. Also: - FiboaBaseConverter: keep the determination:datetime column that `use_variant_as_determination` adds; it was removed again as unlisted (affected dk, hr). - be_vlg: drop plots without a crop code (one in 2023 failed validation), take the determination date from the variant year, add the 2026 edition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ivorbosloper
changed the title
fiboa publish for catalog-driven publication
…note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cz_2023.csv maps the code list of the 2023 release. The 2019-2022 editions declare 120 codes it never saw — 11.4% of their rows, fallow and ware potatoes the largest — so they arrived with no HCAT at all, against 0.0-0.9% in the later editions. The supplement table fills them in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
The 2026 edition lists one declaration twice, once per land block, with the whole declaration's geometry both times — the same polygon under the same ZAKRES_ID. Keep the first of each such repeat. A repeat whose shape differs is not this case and still fails the identity check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
…e grassland data.slovensko.sk publishes "Hranice užívania" once per campaign; the converter read a rolling download instead, so the single edition was labelled by the year it was fetched and held the 2024 file. Take the per-campaign datasets, 2018 through 2026, and date each edition by its campaign. The portal also has 2016 and 2017, but those two are 7-Zip archives the downloader cannot extract. The 2018 release names three columns differently (KDIEL, VYMERA_KD, PORADOVE_C) and every release up to 2021, plus 2025, writes the crop in PLODINA_NA rather than PLODINA. The register writes a no-break space in the three permanent-grassland names, where the crop table has an ordinary one, so they matched nothing: 102,835 fields that the table does have a row for. Normalising the whitespace on both sides drops the unmapped share of 2018-2026 from 7% to 0.5%, and the 75 names now added to the table take it to zero. crop:code was computed and then dropped, as it is not among the published columns and the table is keyed on the crop name; drop the computation too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
…ervice Three things every REST converter needs, each of which es_ib or es_cb had a copy of: - a joined layer qualifies every field (SIGPAC_FOGAIBA.DN_OID), so a converter's `columns` match nothing; the prefixes are stripped on the way out of get_data, first table winning, which is the one with the geometry - an edition may live in a service of its own, so a variant whose value is a URL names the service to read it from - the id-bound query is the one sorted query left, and the Balearic proxy answers two in three with a 502; it is retried five times Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
The yearly layers moved out of GOIB_SIGPAC_IB into GOIB_SIGPAC_HISTORIC_IB, which keeps 2022-2025; what is left in the first service is one current-state layer, ahead of the historic newest (maig against gener 2026). Each variant names the service it comes from. The years before 2022 are withdrawn. Both services join their parcels to the municipality and land-use tables, so their fields arrive table-qualified; the mixin strips that. The snapshot month they carry becomes determination:datetime through column_migrations — the current service writes "maig 2026", the historic ones "Febrer2024.0". `provider` also pointed at Navarra, a copy-paste between the Spanish converters. The converter had no test. It has one page of each service now, 100 features as the service answers them, so the qualified names and both spellings of the snapshot month are covered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
…ounds The Balearic proxy answers two requests in three with a 502, and the service metadata, the field probe and the id bounds are each asked once per run, so one refusal ends the conversion before a page is fetched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
Five refusals in a row are a one-in-eight event on a proxy that answers two in three with a 502, and one run of es_ib hit it. Eight attempts with a capped backoff is about two minutes of tolerance, and each retry says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
The one-row probe that discovers a joined layer's qualified key asked "where=1=1", which the Balearic service refuses on its joined views — eight attempts, eight 502s, no page fetched. The layer's own metadata lists its fields and is a cheap call every layer answers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
A page is requested hundreds of times per edition, so a server refusing one is normal rather than fatal — es_ib got 178 pages into its 2022 edition before one 502 ended the run. Neither a broken download nor an Esri error body survives as a cached file, and the page is asked for again, eight times over about two minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
DE-SH: bring the converter and its fixture up to main's version
Nine converters publish field blocks — a reference parcel bounded by permanent features, which several farmers and crops can share — and five of them were titled "Field boundaries for ...", the same as the crop-field collections beside them. Their own descriptions explain the difference, but a reader scanning a list of titles could not tell them apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
Sweden was never published: the converter knew 2023 and 2024, and the campaign is a CQL filter on one WFS layer, so every year the service answers is a variant. It answers 2015 through 2025, ~1.14 to 1.21 million fields each; 2014 and 2026 return nothing. The http URL redirects to https, which is what the first pass stumbled over. Bulgaria asked for VectorDataSet:Arable_Land_2024, and the GeoServer has no such layer: it publishes VectorData:Agricultural_Land_<year> for 2021-2025 (226,592 blocks in 2025). Beside them are Physical_Blocks_<year>, the same blocks including forest, urban and roads, 1.5 million in 2025 — the agricultural subset is the one this converter wants. That layer carries USAGECODE and the usage in Bulgarian and English, where the converter invented a crop code by numbering a CSV, and it carries no area, so the blocks are measured from their own UTM 35N geometry. Its names arrive as mojibake because GeoServer writes the charset into a .cst file, which GDAL does not read, so UTF-8 is forced. The provider was "Ministry of Health"; mzh is the Ministry of Agriculture and Food. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
They name the block and its usage in one field, ELGIDENT = <settlement>-<block>-<usage>, where 2023 onwards name the block in PHBIDENT and the usage in USAGECODE — so the first build stopped at "none of the columns mapped to 'id' is in this source". They also publish an area (AREA, AREA1 in 2022) that the later editions do not, 2021 truncates the Bulgarian name to ten characters, and 2022 drops it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
A block is listed once per usage, and sometimes twice for the same usage — 42 blocks of the 226,592 in 2025, 40 of them with the same code — so it cannot be the id. It is published as block_id, with the row position identifying the polygon. The 2021 and 2022 editions carry the usage inside ELGIDENT, so their block is the part before it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
PRIA's GSAA WFS publishes the crop as free text and no crop code anywhere in the layer, which is why ee has crop:name and no crop:code — hcat:code is derived from the name. It does carry taotletud_maakasutus, the requested land use: arable, permanent grassland, restored grassland, permanent crops, black fallow. Every edition from 2016 has it. The converter had no test either. It has one now, on a hundred features of the 2024 layer as the service answers them. Catalog issue fieldsoftheworld/harmonized-field-data-catalog#17. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
16 of the 165,244 rows of 2016 share a parcel id, one of them seven times. It is published as parcel_id, with the row index identifying the field in an edition where it repeats — the same shape as fr. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
The code list is ours, at https://fiboa.org/code/ee/ee.csv: a four-digit number per Estonian crop name, frozen once published. 44 names that appear in the editions were missing from the table and are mapped to HCAT as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
EPSG defines SWEREF99 TM northing first, and Jordbruksverket's shapefile honours that: AXIS["Northing", NORTH] before AXIS["Easting", EAST], with the coordinates written that way. GDAL reads them as written, GeoParquet stores x, y, and everything after that point reads such a file as (lat, lon) — the published Swedish tiles and STAC bbox put the country in the Indian Ocean, at lon 60-69, lat 1-6. The declaration alone is not enough to act on, because most sources in a lat-lon-ordered CRS (EPSG:4258, 4326) still store lon, lat — the Spanish fixture does. So the coordinates have to disagree with the CRS's own area of use as well: only a projected CRS that says northing first, whose x values fall outside its easting range and inside its northing range, is flipped. Checked against every published collection: se is the only one wrong. si and hr are in authority-order CRSs too, but their sources deliver easting first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
localId is the LPIS parcel reference and repeats where a parcel is declared more than once — 214 of the 1,027,438 rows of 2022, one of them four times — so the 2022 edition would not convert at all. It is published as parcel_id, and the GML's own feature id, which is unique by construction, is the id. The source publishes no area, so the parcels had none: `metrics:area` is computed from the geometry now, as several other converters do. Median parcel 2.7 ha in the 2024 edition. The department's bucket serves GSAA_2022 through GSAA_2024; GSAA_2017 to GSAA_2021 answer 403 (checked 2026-09-13), which is recorded where the variants are built. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
Carries the converter from split/lv-backfill so the catalog can build the editions; the PR against main holds the same change.
publish-portolan still had the bbox-grid de_sl, which covered part of the state, and no de_sl_block at all. main's versions page through the WFS: 54,038 and 62,500 features, both verified against the live service today.
The Saarland WFS refuses a request without a User-Agent, which the downloader only sends from 0.2.18 (vecorel/cli#30): the same page is 155 kB on 0.2.18 and a FileNotFoundError on 0.2.16. main gets this through #280. Every converter built from this branch now runs under 0.2.18's checks: bounded row drops, required properties from the declared schemas, and the id uniqueness warning.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We will not merge this branch. It's a working branch to publish many datasets to source-coop and the harmonized data catalog. We will pluck commits from here to smaller PRs