[Commerce_NTIA] Modified preprocess.py script

[Krishnam24maheshwari]

This PR updates the Commerce NTIA import pipeline (statvar_imports/ntia_internet_use_survey/commerce_ntia/). It refactors the download and preprocessing scripts with robust error handling and headers, introduces hermetic unit tests, registers node_mcf and output counters in manifest.json, adds data validation checks, and fixes documentation casing.

Key Changes

Preprocessing (preprocess.py)

• Robust Download (HEADERS & Validation): Added HTTP browser headers (User-Agent, Accept) to download_file() to mimic a standard Chrome browser request and prevent government websites from blocking automated Python scripts.
• Added explicit error checking (not success, file existence, and os.path.getsize(INPUT_FILE) == 0) with logging.fatal() so empty or failed downloads fail immediately with a clear error message.
• CSV Encoding Support (encoding='utf-8-sig') : Updated pd.read_csv() to use encoding='utf-8-sig' to automatically strip invisible Byte Order Mark (\ufeff) characters often found in Windows/Excel CSV exports, preventing column name corruption (e.g., reading "\ufeffdataset" instead of "dataset").
• Universe & Variable Age Resolution (_AGE_RESOL_MAP) : Explicitly maps and preserves universeAgeResol and variableAgeResol (CivilPerson for isPerson, Adult for isAdult) across both general and age-specific subsets so StatisticalVariables retain appropriate demographic resolution (Civilian, Age).
• Replaced four repetitive ternary lambda functions across df1 and df2 with a centralized dictionary constant (_AGE_RESOL_MAP) and vectorized Pandas .map(_AGE_RESOL_MAP) for cleaner, more maintainable code.

• Targeted Age Prefix Exclusion:

• Refined the column filter for general survey data (ntia-data.csv) from col.startswith('age') to explicitly check for the 5 age-bracket prefixes ('age314', 'age1524', 'age2544', 'age4564', 'age65p').
• Prevents overly broad filtering that could accidentally drop valid non-age variables starting with "age" (such as agency or ageGroup).
• Directory Safety, Edge Cases & Code Formatting: Moved directory creation (os.makedirs(INPUT_DIR, exist_ok=True)) inside preprocess_data() to ensure safe directory handling without import-time filesystem side effects during unit testing.
• Added an early-return safeguard in move_column_left() when column_to_move == target_column.
• Cleaned up imports, unused arguments (del argv), error handling (logging.fatal()), and standard code formatting.

Validation Rules Summary

1. check_deleted_records_percent: Ensures deleted observation percentage across the import does not exceed 0.1%.
2. check_active_survey_wave_count: Verifies that all active StatVars (1935 in input0, 227 in input1) have MaxDate >= 2023-11, using demographic conditions (age_svs = 0 vs age_svs = total_svs) to ensure mutual exclusivity.
3. check_statvar_min_max_date: Verifies that no StatVar regresses below the earliest historical wave (2013-07).

Test Run Links (Head Commit: 2026_09_21T00_03_54_483382_07_00)

• input0 validation: https://storage.mtls.cloud.google.com/datcom-import-test/statvar_imports/ntia_internet_use_survey/commerce_ntia/Commerce_NTIA/2026_09_21T01_35_00_096847_07_00/input0/validation/validation_output.csv
• differ0 summary: https://storage.mtls.cloud.google.com/datcom-import-test/statvar_imports/ntia_internet_use_survey/commerce_ntia/Commerce_NTIA/2026_09_21T01_35_00_096847_07_00/input0/validation/differ_summary.json
• input1 validation: https://storage.mtls.cloud.google.com/datcom-import-test/statvar_imports/ntia_internet_use_survey/commerce_ntia/Commerce_NTIA/2026_09_21T01_35_00_096847_07_00/input1/validation/validation_output.csv
• differ1 summary: https://storage.mtls.cloud.google.com/datcom-import-test/statvar_imports/ntia_internet_use_survey/commerce_ntia/Commerce_NTIA/2026_09_21T01_35_00_096847_07_00/input1/validation/differ_summary.json

[gemini-code-assist]

Code Review

This pull request introduces unit and regression tests, adds validation configuration, and refactors the preprocessing script for the Commerce NTIA import, including better error handling and output counters. The code reviewer recommends specifying exact MCF files in the manifest instead of using wildcards to prevent cross-contamination, removing a redundant directory creation call, and refactoring nested lambda functions into a cleaner dictionary mapping.

[abhishekjaisw]

Review scope

• Target: PR #2222 (e780f0d0 vs 0226108d)
• Reviewed: manifest.json, validation_config.json, preprocess.py, commerce_ntia_test.py, README.md
• Skipped: None

Summary

The core fix for b/552394335 (adding browser HEADERS and fail-fast download verification in preprocess.py) is verified and succeeded in Cloud Batch test job commerce-ntia-krishnamm-20260914-075721 (datcom-infosys-dev, us-east1, SUCCEEDED in 377.93s). However, inspection of the PR diff and GCS test artifacts (gs://datcom-import-test/.../2026_09_14T01_00_19_735237_07_00/) revealed 1 P1 and 3 P2 issues (plus 1 P3 cleanup) that should be addressed before merging.

Positive findings

• statvar_imports/ntia_internet_use_survey/commerce_ntia/preprocess.py:39 - Resilient HTTP request headers for federal portal downloads ✓
  • Finding: Good - Supplies standard browser User-Agent and Accept headers to download_file(), resolving upstream HTTP 403 Forbidden rejections from ntia.gov.
• statvar_imports/ntia_internet_use_survey/commerce_ntia/preprocess.py:107 - Explicit download verification before transformation ✓
  • Finding: Good - Verifies success, os.path.exists(INPUT_FILE), and os.path.getsize(INPUT_FILE) > 0 before invoking preprocess_data().
• statvar_imports/ntia_internet_use_survey/commerce_ntia/manifest.json:16 - Counter generation and retention configured ✓
  • Finding: Good - Adds --output_counters to stat_var_processor.py and retains "counters/*.csv" under source_files.

Coverage

File	Status	Result
statvar_imports/ntia_internet_use_survey/commerce_ntia/manifest.json	Reviewed	One P1 finding, one P2 finding
statvar_imports/ntia_internet_use_survey/commerce_ntia/validation_config.json	Reviewed	One P2 finding
statvar_imports/ntia_internet_use_survey/commerce_ntia/preprocess.py	Reviewed	One P2 finding, one P3 finding
statvar_imports/ntia_internet_use_survey/commerce_ntia/commerce_ntia_test.py	Reviewed	No actionable findings
statvar_imports/ntia_internet_use_survey/commerce_ntia/README.md	Reviewed	No findings