oecd_regional_education |Increasing deletion threshold

[Ashwani725]

The PR is raised to Allow up to 5% deleted records in OECD regional education validation.

Root Cause

OECD transitioned to dataflow 2.5 (OECD.CFE.EDS:DSD_REG_EDU@DF_ATTAIN(2.5)), introducing Eurostat NUTS 2024 regional restructuring (retiring obsolete territorial units such as PT16, PT17, NL31, and NL33) and pruning historical zero-value estimates. This resulted in 9,531 legitimate record deletions (~3.90% of total observations).

##Test Result:
Test Cloud batch
Test GCS Bucket

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request updates the OECD regional education import manifest to include a node MCF file and a validation configuration file, and introduces a new validation configuration file to check the percentage of deleted records. The reviewer noted a mismatch in the validation configuration where the rule description specifies a 5% threshold, but the parameter is set to 4.

[saanikaaa]

pls remove this file

[Ashwani725]

File Removed.

[saanikaaa]

I see changes are done in preprocess.py, can you pls explain the rational behind those changes

[Ashwani725]

I see changes are done in preprocess.py, can you pls explain the rational behind those changes

The changes in preprocess.py were made to optimize performance and handle the new OECD SDMX Dataflow 2.5 (OECD.CFE.EDS:DSD_REG_EDU@DF_ATTAIN(2.5)) raw CSV format (~1.08M rows and 36 columns) before passing it to stat_var_processor.py:

1. Drastic Execution Time Reduction (~6 hrs / >2 hrs on Batch down to ~12 mins):
   • Without pre-filtering and place pre-resolution, stat_var_processor.py had to parse all 1,080,385 raw rows across 36 columns and attempt place-reconciliation API lookups on ~806k unmapped/Standard Error rows, which caused execution time to take ~6 hours (and 7,561s / >2 hours on Cloud Batch).
   • By pre-filtering invalid rows and pre-resolving REF_AREA to dcid: in memory in preprocess.py, script_execution_time dropped to 588.6s (~9.8 mins) and total end-to-end Cloud Batch execution_time dropped to 703s (~11.7 mins) — well under 2 hours (>90% reduction).
2. Filter out Standard Error (SE) and empty rows:
   • The raw SDMX CSV includes both estimate values and standard error rows (STATISTICAL_OPERATION == 'SE'), as well as rows with blank OBS_VALUE. preprocess.py drops 806,350 non-observation/unmapped rows so only the 274,035 valid observation rows and the 7 required columns (REF_AREA, TIME_PERIOD, UNIT_MULT, SEX, Education level, AGE, OBS_VALUE) are processed by stat_var_processor.py.
3. Pre-resolve REF_AREA codes and log unmapped places:
   • Maps REF_AREA codes against oecd_regional_education_places_resolved.csv (including the newly added NUTS 2024 regions) directly to dcid: values, and logs any unmapped region codes to counters/unresolved_places.csv for observability.
4. Preserve raw downloaded source file & fail-fast validation:
   • Generates oecd_regional_education_data.csv while retaining the original downloaded file (A.........) in gcs_output/source_files/ (instead of os.rename() deleting it), and raises explicit exceptions (FileNotFoundError / ValueError) if the source file is missing or empty to prevent silent data drops.