DE-SH: bring the converter and its fixture up to main's version - #229
Conversation
publish-portolan still carries the pre-fiboa#227 de_sh.py, so anyone rebuilding de_sh from this branch reproduces all three bugs fiboa#223/fiboa#227 fixed: the 2023 archive matches no GDAL driver, fachguelti reaches the STAC step as DD.MM.YYYY text, and the 2025/2026 upper-case spellings silently drop determination:datetime and metrics:area while still validating. The 2026-shaped test fixture comes with it. fiboa#228 brings the expected_columns delivery assertion to this branch, and an assertion whose fixture is still 2024-shaped cannot bite -- 2024 already spells the columns the way the converter maps them, so nothing exercises COLUMN_RENAMES or the decimal-comma area. That combination is exactly what hid the bug in the first place. main has had both since 2026-09-08 but has not been merged into publish-portolan; afterwards both files are identical on the two branches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All four editions were rebuilt from one converter revision and uploaded, so the checksums here describe the bytes now in the bucket. 2026 was rebuilt because the published edition was missing metrics:area and determination:datetime; the rebuild is byte-identical to it across all 194,503 geometries and every column it had. latest/ is a hard link to 2026, so the stable path and the newest edition are the same bytes again. Converter is fiboa/cli#227 on main, with fiboa/cli#229 bringing publish-portolan up to it.
m-mohr
left a comment
m-mohr
left a comment
There was a problem hiding this comment.
@haithcoatj We discussed this today in a meeting with Ivor, Abass and me. We will extract the converter chanages from #214 into separate smaller PRs that can be reviewed more easily, and then close #214. So opening PRs that merge into #214 will eventually also close. Thus, this PR should extract all relevant changes for the DE-SH converter into this or a separate PR so that it can be reviewed separately and merged into main.
|
@ivorbosloper Wasn't this merged into the wrong branch? |
|
@m-mohr Nothing is stranded here: this PR moved code from main into the working branch, not the other way round. #227 had already fixed The one de_sh change that did originate on the working branch is its title — The split is otherwise going as agreed: 40-odd per-dataset PRs against main, and #214 closes when they have landed. Worth knowing: three of the structural ones (#264, #265, #266) are already closed, because vecorel-cli 0.2.18 carries what they proposed — #280 bumps the pin and #281 is what remains of the per-file converter. |
|
@ivorbosloper If I tag you with a specific question, I think I'd prefer a human answer. I can ask the AI myself, which obviously usually just says that it was right just from the beginning.... It was a nice world when a human just responded with a sentence and not an AI with a whole book.... |
@m-mohr You are right, it got out of control. I merely asked Claude to help me analyse this and this response got posted in my name without me noticing. I will try to limit the chattyness and check some of the responses and actions myself 🙏 |
3 participants
publish-portolan still has the pre-#227 de_sh.py, so anyone rebuilding de_sh from this branch reproduces the three bugs #227 fixed. The commit message has the details. main has had the fix since Sep 8 but hasn't been merged in here, and this is where the converter work is happening.
Brings de_sh.py and its test fixture to main's versions, unchanged. Both are byte-identical across the two branches afterwards, so a later merge of main is a no-op for them.
The fixture is worth calling out on its own: the one on this branch is named Feldbloecke_2026_GPKG.zip but holds layer fb_2024 with 2024's mixed-case columns and the area as a Real. A file named for one edition containing another's data is wrong regardless of what any test does with it.
On coverage, measured rather than assumed. With the fixture in place but #228's expected_columns assertion not yet here, breaking parse_decimal fails and breaking parse_date fails. Breaking COLUMN_RENAMES still passes, because the only substantive check is
assert (df["metrics:area"] > 10).all()sitting behindif "metrics:area" in df.columns— so dropping the column skips the assertion instead of failing it. That's the same presence guard that hid the original bug, and it closes when #228 lands.Clean pick: the imports and the
{url: ["*.gpkg"]}variants form already exist here (cz.py, es_cl.py), and it satisfies this branch's newer guards including _require_unique_ids — the id source is unique and non-null in all four editions, measured on the real archives at 198,614 / 197,673 / 195,747 / 194,503 rows. tests/test_convert.py is untouched, so this and #228 change disjoint files and can land in either order.