oecd_regional_education |Increasing deletion threshold

statvar_imports/oecd/regional_education/README.md

@@ -1,93 +1,79 @@
 ### OECD Regional Education
-### 1. Import Overview
-This project processes and imports education-related data from the OECD (Organisation for Economic Co-operation and Development), focusing on regional education statistics.
-
-`Source URL`: https://stats.oecd.org/Index.aspx?DataSetCode=REGION_EDUCAT
-
-`Import Type`: Script-driven download via download_script.py.
-
-`Source Data Availability`: Data is available from 2000 to the latest year, including monthly data for each year and the latest month.
-
-`Type of Place`: Country, State, and Administrative Area (OECD countries).
-
-`StatVars`: Primarily related to Men, Women, and both sexes' education qualification data.
-
-### Notes: 
-The dataset primarily includes educational attainment data for two distinct age groups: 25 to 34 years and 25 to 64 years.
-
-The data covers countries listed within the OECD member nations.
-
-Mean value has been taken and standard deviation values are ignored.
-
-### 2. Preprocessing Steps
-Before ingestion, the following preprocessing is done:
-
-**Input files:**
-
-`oecd_regional_education_data.csv:` Raw input data (created by download_script.py in the input folder)
-
-`oecd_regional_education_pvmap.csv:` Property-value mapping
-
-`oecd_regional_education_metadata.csv:` StatVar metadata (used by stat_var_processor.py)
-
-`oecd_regional_education_places_resolved.csv:` place_resolved.csv 
-
-### Transformation pipeline:
-
-The download.py creates an input folder for raw files and an output folder for processed results.
-
-Data is processed using stat_var_processor.py.
-
-No transformations are performed; the raw data is used as is directly from the downloaded source.
-
-### Data Quality Checks:
-
-Linting is performed using the DataCommons import tool JAR.
-
-### 3. Autorefresh Type
-
-***Full Automation***
-
-***Steps:***
-
-The import job runs automatically in the cloud every two weeks, specifically at 10:00 AM UTC on the 1st and 15th of each month (based on `cron schedule 0 10 1,15 * *`).
-
-It executes the download_util_script.py to retrieve the latest data from the OECD SDMX API endpoint: https://sdmx.oecd.org/public/rest/data/OECD.CFE.EDS,DSD_REG_EDU@DF_ATTAIN,/A.........?dimensionAtObservation=AllDimensions&format=csvfilewithlabels.
-
-Subsequently, the stat_var_processor.py is run to process the downloaded raw data, generating the final oecd_regional_education.csv and oecd_regional_education.tmcf files.
-
-The generated **output files**: `(oecd_regional_education.csv and oecd_regional_education.tmcf)` are prepared for ingestion.
-
-### Note: This pipeline is fully automated.
 
+### 1. Import Overview
+This project processes and imports regional educational attainment statistics from the OECD (Organisation for Economic Co-operation and Development) via the OECD SDMX REST API (Dataflow 2.5: `OECD.CFE.EDS:DSD_REG_EDU@DF_ATTAIN(2.5)`).
+
+- **Source URL**: `https://stats.oecd.org/Index.aspx?DataSetCode=REGION_EDUCAT`
+- **SDMX API Endpoint**: `https://sdmx.oecd.org/public/rest/data/OECD.CFE.EDS,DSD_REG_EDU@DF_ATTAIN,/A.........?dimensionAtObservation=AllDimensions&format=csvfilewithlabels`
+- **Import Type**: Automated download via `download_util_script.py` followed by preprocessing in `preprocess.py` and StatVar processing via `stat_var_processor.py`.
+- **Source Data Availability**: Annual regional data from 2000 to the latest available year.
+- **Type of Place**: Country, State, and NUTS / OECD Territorial Level regions (`TL2` / `TL3`).
+- **StatVars**: Educational attainment by age group (`25 to 34 years` and `25 to 64 years`), sex (`Male`, `Female`, and `Total`), and ISCED-2011 education level.
+
+### 2. Preprocessing & Transformation Pipeline
+
+**Input and configuration files:**
+- `gcs_output/source_files/A.........`: Raw CSV downloaded from the OECD SDMX API by `download_util_script.py`.
+- `oecd_regional_education_places_resolved.csv`: Mapping of OECD `REF_AREA` codes (including Eurostat NUTS 2024 regions) to Data Commons `dcid`s.
+- `oecd_regional_education_pvmap.csv`: Property-value mapping for `stat_var_processor.py`.
+- `oecd_regional_education_metadata.csv`: Configuration metadata for `stat_var_processor.py`.
+- `validation_config.json`: Validation rules, including a 5% deleted records threshold (`DELETED_RECORDS_PERCENT`) to accommodate OECD Dataflow 2.5 NUTS 2024 regional boundary restructuring (e.g., retirement of obsolete NUTS 2021 codes `PT16`, `PT17`, `NL31`, `NL33` and replacement with `PT19`–`PT1D`, `NL35`, `NL36`), `MAX_DATE_CONSISTENT`, freshness lag checks, and StatVar coverage checks.
+
+**Transformation steps (`preprocess.py`):**
+1. Loads resolved region mappings (`REF_AREA` -> `dcid`) from `oecd_regional_education_places_resolved.csv` in memory.
+2. Reads the raw downloaded OECD SDMX CSV (`A.........`) from `gcs_output/source_files/` while preserving the original downloaded file for provenance.
+3. Filters out rows with empty `OBS_VALUE` or standard error rows (`STATISTICAL_OPERATION == 'SE'`).
+4. Pre-resolves `REF_AREA` codes to `dcid:` values, extracts the required columns (`REF_AREA`, `TIME_PERIOD`, `UNIT_MULT`, `SEX`, `Education level`, `AGE`, `OBS_VALUE`), and writes the cleaned dataset to `gcs_output/source_files/oecd_regional_education_data.csv`.
+5. Logs any unmapped `REF_AREA` codes to `counters/unresolved_places.csv` for observability.
+
+### 3. Autorefresh Schedule
+
+- **Schedule**: Runs automatically every two weeks at 10:00 AM UTC on the 1st and 15th of each month (`0 10 1,15 * *`).
+- **Pipeline sequence** (configured in `manifest.json`):
+  1. `download_util_script.py`: Downloads the raw SDMX CSV into `gcs_output/source_files/`.
+  2. `preprocess.py`: Filters rows and resolves regional place codes into `gcs_output/source_files/oecd_regional_education_data.csv`.
+  3. `stat_var_processor.py`: Generates `output/oecd_regional_education.csv`, `output/oecd_regional_education.tmcf`, and StatVar MCFs (`output/*.mcf`), plus counters in `counters/oecd_regional_education_counters.csv`.
 
 ### 4. Script Execution Details
-***Script 1:*** download.py (Download Script)
-Usage:
 
-```Bash
-
-python3 download.py
-Purpose: It downloads the latest data from the source.
+All pipeline commands below should be executed from the dataset directory:
+```bash
+cd statvar_imports/oecd/regional_education
 ```
 
-***Script 2:*** stat_var_processor.py (Data Processing)
-Usage:
-
-**General Usage:**
-
-(Processing from current import folder data/statvar_imports/oecd/regional_education):
+**Step 1: Download raw data**
+```bash
+python3 ../../../util/download_util_script.py \
+  --download_url='https://sdmx.oecd.org/public/rest/data/OECD.CFE.EDS,DSD_REG_EDU@DF_ATTAIN,/A.........?dimensionAtObservation=AllDimensions&format=csvfilewithlabels' \
+  --output_folder=gcs_output/source_files
+```
 
-```Bash
+**Step 2: Run preprocessing**
+```bash
+python3 preprocess.py
+```
 
-python3 ../../../tools/statvar_importer/stat_var_processor.py --input_data=input/oecd_regional_education_data.csv --pv_map=oecd_regional_education_pvmap.csv --config_file=oecd_regional_education_metadata.csv --output_path=output/oecd_regional_education
+**Step 3: Process StatVars and observations**
+```bash
+python3 ../../../tools/statvar_importer/stat_var_processor.py \
+  --input_data=gcs_output/source_files/oecd_regional_education_data.csv \
+  --pv_map=oecd_regional_education_pvmap.csv \
+  --config_file=oecd_regional_education_metadata.csv \
+  --places_resolved_csv=oecd_regional_education_places_resolved.csv \
+  --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \
+  --output_path=output/oecd_regional_education \
+  --output_counters=counters/oecd_regional_education_counters.csv
 ```
-(Processing from statvar_importer folder data/tools/statvar_importer):
 
-```Bash
+### 5. Running Unit Tests
 
-python3 stat_var_processor.py --input_data=input/oecd_regional_education_data.csv --pv_map=oecd_regional_education_pvmap.csv --config_file=oecd_regional_education_metadata.csv --output_path=output/oecd_regional_education
-Purpose: Generates StatVar MCF, cleans observation CSV, TMCF.
+Run the unit test suite from this directory:
+```bash
+python3 -m unittest preprocess_test.py
+```
 
+Or from the root of the repository:
+```bash
+python3 -m unittest statvar_imports/oecd/regional_education/preprocess_test.py
 ```
 

statvar_imports/oecd/regional_education/manifest.json

@@ -6,22 +6,25 @@
                 "support@datacommons.org"
             ],
             "provenance_url": "https://stats.oecd.org/Index.aspx?DataSetCode=REGION_EDUCAT",
-            "provenance_description": "dataset contains commodity price and its index value for monthly and annual",
+            "provenance_description": "dataset contains regional educational attainment statistics for OECD regions and cities",
             "scripts": [
                 "../../../util/download_util_script.py --download_url='https://sdmx.oecd.org/public/rest/data/OECD.CFE.EDS,DSD_REG_EDU@DF_ATTAIN,/A.........?dimensionAtObservation=AllDimensions&format=csvfilewithlabels' --output_folder=gcs_output/source_files",
                 "preprocess.py",
-                "../../../tools/statvar_importer/stat_var_processor.py --input_data=gcs_output/source_files/oecd_regional_education_data.csv --pv_map=oecd_regional_education_pvmap.csv --config_file=oecd_regional_education_metadata.csv --places_resolved_csv=oecd_regional_education_places_resolved.csv --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --output_path=output/oecd_regional_education"
+                "../../../tools/statvar_importer/stat_var_processor.py --input_data=gcs_output/source_files/oecd_regional_education_data.csv --pv_map=oecd_regional_education_pvmap.csv --config_file=oecd_regional_education_metadata.csv --places_resolved_csv=oecd_regional_education_places_resolved.csv --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --output_path=output/oecd_regional_education --output_counters=counters/oecd_regional_education_counters.csv"
             ],
             "import_inputs": [
                 {
                     "template_mcf": "output/oecd_regional_education.tmcf",
-                    "cleaned_csv": "output/oecd_regional_education.csv"
+                    "cleaned_csv": "output/oecd_regional_education.csv",
+                    "node_mcf": "output/*.mcf"
                 }
             ],
             "cron_schedule": "0 10 1,15 * *",
+            "validation_config_file": "validation_config.json",
             "source_files": [
-                "gcs_output/source_files/*.csv"
+                "gcs_output/source_files/*",
+                "counters/*.csv"
             ]
         }
     ]
-}
+}

statvar_imports/oecd/regional_education/oecd_regional_education_metadata.csv

@@ -1,14 +1,15 @@
-parameter,value
-url,https://stats.oecd.org/Index.aspx?DataSetCode=REGION_DEMOGR#
-description,Educational Attainment for regions and cities
-#place_type,
-#places_within,"country, AA1,AA2"
-start_date,1998
-end_date,2022
-release_frequency,Yearly
-process,
-comments,"The source for education data is from 1993 to 2022, unable to selet all the required years to process. 
-Path to data - URL - Regions and Cities - Regional Statistics - Regional Education - Educational Attainment"
-output_columns,"observationAbout,observationDate,variableMeasured,value"
-header_rows,1
-word_delimiter,""""""
+parameter,value
+url,https://stats.oecd.org/Index.aspx?DataSetCode=REGION_EDUCAT
+description,Educational Attainment for regions and cities
+#place_type,
+#places_within,"country, AA1,AA2"
+start_date,2000
+end_date,2025
+release_frequency,Yearly
+process,
+comments,"The source for education data is from 2000 to 2025. Path to data - URL - Regions and Cities - Regional Statistics - Regional Education - Educational Attainment"
+output_columns,"observationAbout,observationDate,variableMeasured,value"
+header_rows,1
+word_delimiter,""""""
+reconcile_nodes,
+mapped_columns,"4,5,6"

statvar_imports/oecd/regional_education/oecd_regional_education_places_resolved.csv

@@ -119,7 +119,7 @@ BG41,nuts/BG41,South West,Bulgaria
 BG42,nuts/BG42,South Central,Bulgaria
 CAN,country/CAN,Canada,Canada
 CA10,wikidataId/Q2003,Newfoundland and Labrador,Canada
-CA11, wikidataId/Q1979,Prince Edward Island,Canada
+CA11,wikidataId/Q1979,Prince Edward Island,Canada
 CA12,wikidataId/Q1952,Nova Scotia,Canada
 CA13,wikidataId/Q1965,New Brunswick,Canada
 CA24,wikidataId/Q176,Quebec,Canada
@@ -683,6 +683,8 @@ NL31,nuts/NL31,Utrecht,Netherlands
 NL32,nuts/NL32,North Holland,Netherlands
 NL33,nuts/NL33,South Holland,Netherlands
 NL34,nuts/NL34,Zeeland,Netherlands
+NL35,nuts/NL35,Utrecht,Netherlands
+NL36,nuts/NL36,South Holland,Netherlands
 NL41,nuts/NL41,North Brabant,Netherlands
 NL42,nuts/NL42,Limburg,Netherlands
 NZ23,wikidataId/Q657004,Canterbury,New Zealand
@@ -736,6 +738,11 @@ PT15,nuts/PT15,Algarve,Portugal
 PT16,nuts/PT16,Central Portugal,Portugal
 PT17,nuts/PT17,Metropolitan area of Lisbon,Portugal
 PT18,nuts/PT18,Alentejo,Portugal
+PT19,nuts/PT19,Centre,Portugal
+PT1A,nuts/PT1A,Greater Lisbon,Portugal
+PT1B,nuts/PT1B,Setúbal Peninsula,Portugal
+PT1C,nuts/PT1C,Alentejo,Portugal
+PT1D,nuts/PT1D,West and Tagus Valley,Portugal
 PT20,nuts/PT20,Autonomous Region of the Azores,Portugal
 PT30,nuts/PT30,Autonomous Region of Madeira,Portugal
 ROU,country/ROU,Romania,Romania
@@ -1036,4 +1043,4 @@ US51,geoId/51,Virginia,United States
 US53,geoId/53,Washington,United States
 US54,geoId/54,West Virginia,United States
 US55,geoId/55,Wisconsin,United States
-US56,geoId/56,Wyoming,United States
+US56,geoId/56,Wyoming,United States