[Zurich] Update Zurich_Population and Zurich_Population_Number_Of_Company_Workplace_Employees

statvar_imports/zurich/README.md

@@ -1,29 +1,40 @@
 1. import_name": "Zurich_Population_Number_Of_Company_Workplace_Employees"
 
 2. Import Overview
-Zurich population data at Province and City Level.
-Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/wir_2552_wiki.csv)
+Number of companies, workplaces and employees in Zurich city at City, District (Kreise), and Quarter (Quartiere) Level.
+Source URL: [BFS WIR STATENT Data](https://data.stadt-zuerich.ch/dataset/bfs_wir_statent_ast_beschaeftigte_vza_rechtsform_betrgr_jahr_od2552)
 Import Type: Fully Autorefresh
-Source Data Availability: 2011 to 2022
+Source Data Availability: 2011 to 2024
 Release Frequency: P1Y
 
-3. Preprocessing Steps (No)
+3. Preprocessing Steps (Yes)
+Generate rollups from the downloaded dataset:
+python3 wir_2552_wiki/generate_rollups.py
+
+Rollup Logic & Explanation (`wir_2552_wiki/generate_rollups.py`):
+- **Why Filter on Value `'0'` (`RechtsformSort == 0` & `BetriebsgrSort == 0`)**:
+  - In the upstream dataset (`WIR255OD2552.csv`), category code **`0` represents the official aggregate total row** across all sub-categories: `RechtsformSort == 0` is *"Alle Rechtsformen"* (all legal forms combined) and `BetriebsgrSort == 0` is *"Alle Betriebsgrössen"* (all company sizes combined). Non-zero values (`1`, `2`, etc.) represent granular sub-category breakdowns (e.g., public-law vs. private-law entities, micro-businesses `<10` vs. small businesses `10–49`).
+  - **Matches Target StatVar Schema**: The Data Commons StatVars emitted by this import (`Count_Company`, `Count_Person_Employed`, `Count_Person_Employed_Female`, `Count_Person_Employed_Male`, `Count_Person_FullTimeEmployee`, etc.) represent overall totals per place and year without legal form or company size constraints.
+  - **Prevents Conflicting Duplicate Observations**: Retaining non-zero breakdown rows would cause `stat_var_processor.py` to map multiple sub-category rows to the exact same `(place, year, StatVar)` tuple, causing observation collisions.
+  - **Avoids Undercounting from Suppressed Cells (`'K'`)**: Individual sub-category breakdown rows frequently contain `'K'` (confidentiality suppression for small counts). Summing non-zero breakdown rows manually would undercount true totals, whereas the official `(0, 0)` row provides the complete, authoritative total published by Statistics Zurich.
+- **Unknown Region Exclusion**: Filters out unknown/unassigned region rows (`RaumSort` `990` ["Kreis Unbekannt"] and `999` ["Quartier Unbekannt"]) that do not map to valid geographic places.
+- **Numeric Coercion**: Coerces metric columns (`Arbeitsstaetten`, `AnzBesch`, `AnzBeschW`, `AnzBeschM`, `AnzVZA`, `AnzVZAW`, `AnzVZAM`) to numeric values, converting non-numeric markers (e.g., `'K'` for confidential/suppressed entries) to `NaN` so they are cleanly skipped during StatVar observation generation.
 
 4. Autorefresh Type
 
-Fully Autorefresh:"0 2 29 * * " (Runs at 2:00 AM on the 29th day of every month).
+Fully Autorefresh:"0 2 1,15 * * " (Runs at 2:00 AM on the 1st and 15th day of every month).
 
 5. Script Execution Details
 
-" python3 stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data='../../statvar_imports/zurich/wir_2552_wiki/test_data/wir_2552_wiki_input.csv' --pv_map='../../statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_pvmap.csv' --config_file='../../statvar_imports/zurich/wir_2552_wiki/wir_2552_wiki_metadata.csv' --output_path=../../statvar_imports/zurich/wir_2552_wiki/output/zurich_population_wir_2552_wiki_output  "
+" python3 ../../util/download_util_script.py --download_url=https://data.stadt-zuerich.ch/dataset/bfs_wir_statent_ast_beschaeftigte_vza_rechtsform_betrgr_jahr_od2552/download/WIR255OD2552.csv --output_folder=wir_2552_wiki/input_files && python3 wir_2552_wiki/generate_rollups.py && python3 ../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=wir_2552_wiki/input_files/WIR255OD2552_rollups.csv --pv_map=wir_2552_wiki/wir_2552_wiki_pvmap.csv --config_file=wir_2552_wiki/wir_2552_wiki_metadata.csv --output_columns=observationAbout,observationDate,value,variableMeasured --output_path=wir_2552_wiki/output/zurich_population_wir_2552_wiki --output_counters=wir_2552_wiki/counters/zurich_population_wir_2552_wiki_counters.csv "
 
 #####
 
 
 1. import_name": "Zurich_Population_By_Age"
 
 2. Import Overview
-Zurich population data at Province and City Level.
+Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
 Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3903_age10_wiki.csv)
 Import Type: Fully Autorefresh
 Source Data Availability: 1993 to 2023
@@ -45,29 +56,29 @@ Fully Autorefresh:" 0 6 29 * * " (Runs at 6:00 AM on the 29th of every month).
 1. import_name": "Zurich_Population"
 
 2. Import Overview
-Zurich population data at Province and City Level.
-Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3240_wiki.csv)
+Total population of Zurich city by quarter and year at Quarter (Quartiere) Level.
+Source URL: [BEV324OD3240 Dataset](https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240)
 Import Type: Fully Autorefresh
-Source Data Availability: 1941 to 2023
+Source Data Availability: 1941 to 2025
 Release Frequency: P1Y
 
 3. Preprocessing Steps (No)
 
 4. Autorefresh Type
 
-Fully Autorefresh:" 30 11 29 * * " (Runs at 11:30 AM on the 29th of every month).
+Fully Autorefresh:" 30 11 1,15 * * " (Runs at 11:30 AM on the 1st and 15th of every month).
 
 5. Script Execution Details
 
-" python3 stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data='../../statvar_imports/zurich/bev_3240_wiki/test_data/bev_3240_wiki_input.csv' --pv_map='../../statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_pvmap.csv' --config_file='../../statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_metadata.csv' --output_path=../../statvar_imports/zurich/bev_3240_wiki/output/zurich_population_bev_3240_wiki.csv_output "
+" python3 ../../util/download_util_script.py --download_url=https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240/download/BEV324OD3240.csv --output_folder=bev_3240_wiki/input_files && python3 ../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=bev_3240_wiki/input_files/BEV324OD3240.csv --pv_map=bev_3240_wiki/bev_3240_wiki_pvmap.csv --config_file=bev_3240_wiki/bev_3240_wiki_metadata.csv --output_columns=observationAbout,observationDate,value,variableMeasured --output_path=bev_3240_wiki/output/zurich_population_bev_3240_wiki --output_counters=bev_3240_wiki/counters/zurich_population_bev_3240_wiki_counters.csv "
 
 #####
 
 
 1. import_name": "Zurich_Population_Number_Of_Birth_By_Origin"
 
 2. Import Overview
-Zurich population data at Province and City Level.
+Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
 Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_4031_hel_wiki.csv)
 Import Type: Fully Autorefresh
 Source Data Availability: 1998 to 2023
@@ -89,7 +100,7 @@ Fully Autorefresh:" 45 15 29 * * " (Runs at 3:45 PM on the 29th of every month).
 1. import_name": "Zurich_Population_Number_Of_Birth"
 
 2. Import Overview
-Zurich population data at Province and City Level.
+Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
 Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_4031_wiki.csv)
 Import Type: Fully Autorefresh
 Source Data Availability: 1998 to 2023
@@ -111,7 +122,7 @@ Fully Autorefresh:" 0 20 29 * * " (Runs at 8:00 PM on the 29th of every month).
 1. import_name": "Zurich_Population_By_Origin"
 
 2. Import Overview
-Zurich population data at Province and City Level.
+Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
 Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3903_hel_wiki.csv)
 Import Type: Fully Autorefresh
 Source Data Availability: 1993 to 2023
@@ -134,7 +145,7 @@ Fully Autorefresh:" 15 1 29 * * " (Runs at 1:15 AM on the 29th of every month).
 1. import_name": "Zurich_Population_Number_Of_Birth_By_Sex"
 
 2. Import Overview
-Zurich population data at Province and City Level.
+Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
 Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_4031_sex_wiki.csv)
 Import Type: Fully Autorefresh
 Source Data Availability: 1993 to 2023
@@ -156,7 +167,7 @@ Fully Autorefresh:" 07 19 29 * * " (Runs at 7:07 PM on the 29th of every month).
 1. import_name": "Zurich_Population_By_Sex"
 
 2. Import Overview
-Zurich population data at Province and City Level.
+Zurich population data at City, District (Kreise), and Quarter (Quartiere) Level.
 Source URL: [Zurich country website](https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3903_sex_wiki.csv)
 Import Type: Fully Autorefresh
 Source Data Availability: 1993 to 2023

statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_metadata.csv

@@ -1,14 +1,13 @@
 parameter,value
-url,https://www.stadt-zuerich.ch/content/dam/web/de/politik-verwaltung/statistik-und-daten/linked-open-data/datacommons/bev_3240_wiki.csv
+url,https://data.stadt-zuerich.ch/dataset/bev_bestand_jahr_quartier_od3240
 description,
-place_type,Country
+place_type,City
 #places_within,
 start_date,
 end_date,
 release_frequency,
 process,
 comments,
-output_columns,"observationAbout,observationDate,value,variableMeasured,unit"
+output_columns,"observationAbout,observationDate,value,variableMeasured"
 header_rows,1
-#input_rows,10
-#Import_Name,Zurich_Population
+reconcile_nodes,False

statvar_imports/zurich/bev_3240_wiki/bev_3240_wiki_pvmap.csv

@@ -1,4 +1,37 @@
-key,prop,val,p1,v1
-StichtagDatJahr,observationDate,{Number},,
-AnzBestWir,value,{Number},populationType,Person
-wikiId,observationAbout,wikidataId/{Data},,
+key,prop,val,p1,v1
+StichtagDatJahr,observationDate,{Number},,
+AnzBestWir,value,{Number},populationType,Person
+QuarLang:Rathaus,observationAbout,wikidataId/Q692511,,
+QuarLang:Hochschulen,observationAbout,wikidataId/Q39240,,
+QuarLang:Lindenhof,observationAbout,wikidataId/Q10987378,,
+QuarLang:City,observationAbout,wikidataId/Q1093831,,
+QuarLang:Wollishofen,observationAbout,wikidataId/Q642353,,
+QuarLang:Leimbach,observationAbout,wikidataId/Q691367,,
+QuarLang:Enge,observationAbout,wikidataId/Q648218,,
+QuarLang:Alt-Wiedikon,observationAbout,wikidataId/Q433012,,
+QuarLang:Friesenberg,observationAbout,wikidataId/Q693357,,
+QuarLang:Sihlfeld,observationAbout,wikidataId/Q370104,,
+QuarLang:Werd,observationAbout,wikidataId/Q531899,,
+QuarLang:Langstrasse,observationAbout,wikidataId/Q1805410,,
+QuarLang:Hard,observationAbout,wikidataId/Q870084,,
+QuarLang:Gewerbeschule,observationAbout,wikidataId/Q693413,,
+QuarLang:Escher Wyss,observationAbout,wikidataId/Q687052,,
+QuarLang:Unterstrass,observationAbout,wikidataId/Q656446,,
+QuarLang:Oberstrass,observationAbout,wikidataId/Q693483,,
+QuarLang:Fluntern,observationAbout,wikidataId/Q693269,,
+QuarLang:Hottingen,observationAbout,wikidataId/Q693454,,
+QuarLang:Hirslanden,observationAbout,wikidataId/Q476940,,
+QuarLang:Witikon,observationAbout,wikidataId/Q392079,,
+QuarLang:Seefeld,observationAbout,wikidataId/Q692773,,
+QuarLang:Mühlebach,observationAbout,wikidataId/Q693397,,
+QuarLang:Weinegg,observationAbout,wikidataId/Q693321,,
+QuarLang:Albisrieden,observationAbout,wikidataId/Q80797,,
+QuarLang:Altstetten,observationAbout,wikidataId/Q445711,,
+QuarLang:Höngg,observationAbout,wikidataId/Q455496,,
+QuarLang:Wipkingen,observationAbout,wikidataId/Q678030,,
+QuarLang:Affoltern,observationAbout,wikidataId/Q382903,,
+QuarLang:Oerlikon,observationAbout,wikidataId/Q167179,,
+QuarLang:Seebach,observationAbout,wikidataId/Q276792,,
+QuarLang:Saatlen,observationAbout,wikidataId/Q652455,,
+QuarLang:Schwamendingen-Mitte,observationAbout,wikidataId/Q692728,,
+QuarLang:Hirzenbach,observationAbout,wikidataId/Q693374,,

statvar_imports/zurich/bev_3240_wiki/test_data/bev_3240_wiki_input.csv

@@ -1,25 +1,6 @@
-StichtagDatJahr,AnzBestWir,wikiId
-2023,3343,Q692511
-2022,3330,Q692511
-2021,3275,Q692511
-2020,3317,Q692511
-2019,3307,Q692511
-2018,3267,Q692511
-2017,3266,Q692511
-2016,3226,Q692511
-2015,3197,Q692511
-2014,3236,Q692511
-2013,3194,Q692511
-2012,3164,Q692511
-2011,3140,Q692511
-2010,3099,Q692511
-2009,3110,Q692511
-2008,3051,Q692511
-2007,3079,Q692511
-2006,3097,Q692511
-2005,3081,Q692511
-2004,3115,Q692511
-2003,3149,Q692511
-2002,3128,Q692511
-2001,3103,Q692511
-2000,3109,Q692511
+StichtagDatJahr,QuarSort,QuarLang,AnzBestWir
+2025,11,Rathaus,3252
+2024,11,Rathaus,3260
+2023,11,Rathaus,3343
+2022,11,Rathaus,3330
+2021,11,Rathaus,3275

statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.csv

@@ -1,25 +1,6 @@
-observationAbout,observationDate,value,variableMeasured,unit
-wikidataId/Q692511,2023,3343,dcid:Count_Person,
-wikidataId/Q692511,2022,3330,dcid:Count_Person,
-wikidataId/Q692511,2021,3275,dcid:Count_Person,
-wikidataId/Q692511,2020,3317,dcid:Count_Person,
-wikidataId/Q692511,2019,3307,dcid:Count_Person,
-wikidataId/Q692511,2018,3267,dcid:Count_Person,
-wikidataId/Q692511,2017,3266,dcid:Count_Person,
-wikidataId/Q692511,2016,3226,dcid:Count_Person,
-wikidataId/Q692511,2015,3197,dcid:Count_Person,
-wikidataId/Q692511,2014,3236,dcid:Count_Person,
-wikidataId/Q692511,2013,3194,dcid:Count_Person,
-wikidataId/Q692511,2012,3164,dcid:Count_Person,
-wikidataId/Q692511,2011,3140,dcid:Count_Person,
-wikidataId/Q692511,2010,3099,dcid:Count_Person,
-wikidataId/Q692511,2009,3110,dcid:Count_Person,
-wikidataId/Q692511,2008,3051,dcid:Count_Person,
-wikidataId/Q692511,2007,3079,dcid:Count_Person,
-wikidataId/Q692511,2006,3097,dcid:Count_Person,
-wikidataId/Q692511,2005,3081,dcid:Count_Person,
-wikidataId/Q692511,2004,3115,dcid:Count_Person,
-wikidataId/Q692511,2003,3149,dcid:Count_Person,
-wikidataId/Q692511,2002,3128,dcid:Count_Person,
-wikidataId/Q692511,2001,3103,dcid:Count_Person,
-wikidataId/Q692511,2000,3109,dcid:Count_Person,
+observationAbout,observationDate,value,variableMeasured
+wikidataId/Q692511,2025,3252,dcid:Count_Person
+wikidataId/Q692511,2024,3260,dcid:Count_Person
+wikidataId/Q692511,2023,3343,dcid:Count_Person
+wikidataId/Q692511,2022,3330,dcid:Count_Person
+wikidataId/Q692511,2021,3275,dcid:Count_Person

statvar_imports/zurich/bev_3240_wiki/test_data/zurich_population_bev_3240_wiki_output.tmcf

@@ -1,7 +1,6 @@
-Node: E:bev_3240_wiki_output->E0
-observationAbout: C:bev_3240_wiki_output->observationAbout
-observationDate: C:bev_3240_wiki_output->observationDate
-value: C:bev_3240_wiki_output->value
-variableMeasured: C:bev_3240_wiki_output->variableMeasured
-unit: C:bev_3240_wiki_output->unit
+Node: E:zurich_population_bev_3240_wiki_output->E0
+observationAbout: C:zurich_population_bev_3240_wiki_output->observationAbout
+observationDate: C:zurich_population_bev_3240_wiki_output->observationDate
+value: C:zurich_population_bev_3240_wiki_output->value
+variableMeasured: C:zurich_population_bev_3240_wiki_output->variableMeasured
 typeOf: dcs:StatVarObservation

statvar_imports/zurich/bev_3240_wiki/validation_config.json

@@ -0,0 +1,28 @@
+{
+    "schema_version": "1.0",
+    "rules": [
+        {
+            "rule_id": "check_deleted_records_percent",
+            "description": "Checks that deleted records do not exceed 0.1%.",
+            "validator": "DELETED_RECORDS_PERCENT",
+            "params": {
+                "threshold": 0.1
+            }
+        },
+        {
+            "rule_id": "check_max_date_consistent",
+            "description": "Checks if the MaxDate is the same for all StatVars.",
+            "validator": "MAX_DATE_CONSISTENT",
+            "params": {}
+        },
+        {
+            "rule_id": "check_max_date_freshness",
+            "description": "Verifies that MaxDate meets the expected freshness threshold for the dataset.",
+            "validator": "SQL_VALIDATOR",
+            "params": {
+                "query": "SELECT MAX(MaxDate) >= '2025' AS is_fresh, COUNT(*) AS row_cnt FROM stats",
+                "condition": "is_fresh = TRUE AND row_cnt > 0"
+            }
+        }
+    ]
+}