EIA Imports : Add node_mcf and regenerate golden data for EIA imports - #2224
kartik-s21 wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates several EIA golden data files by sorting the place identifiers alphabetically and reordering the columns in the summary reports. Additionally, it updates manifest.json to include the node_mcf path for multiple import configurations. There are no review comments, and I have no feedback to provide.
…enerate golden data - Update node_mcf to wildcard pattern gcs_output/<DATASET>/*.mcf for all 8 imports to ingest both StatisticalVariable and StatVarGroup MCFs - Add golden_data/*.csv to source_files in manifest.json - Regenerate observation goldens matching top 100k places - Sub-sample golden summary reports to concise representative records adhering to golden guidelines
69d3257 to
22e6470
Compare
rohitkumarbhagat
left a comment
There was a problem hiding this comment.
Please update the PR title, description, and validation configs before merging:
- PR Title & Summary: The current title (
EIA Imports : Add node_mcf and regenerate golden data for EIA imports) and body (Added "node_mcf": "gcs_output/<DATASET>/<DATASET>.mcf" to all 8 EIA imports) still describe the revertednode_mcfaddition rather than the latest commit (5fa327a0), which removesnode_mcfand adds"golden_data/*.csv"tosource_files. - Validation Links: Several validation links in the PR description (
[EIA_Electricity](),[EIA_Petroleum]()) are empty[]()placeholders—please update them with the2026_09_20test run links. - [P2] Date Freshness Validation (
validation_config_EIA_*.json): Sincegolden_summary_report_EIA_*.csvexcludesMaxDateandMaxDatevaries across series insummary_report.csv, please add a scopedSQL_VALIDATORdate freshness check tovalidation_config_EIA_*.json.
| "source_files": [ | ||
| "gcs_output/COAL/COAL.txt" | ||
| "gcs_output/COAL/COAL.txt", | ||
| "golden_data/*.csv" |
There was a problem hiding this comment.
[P3] Documentation & Metadata Hygiene
Finding: The PR title and description still state that node_mcf was added to all 8 EIA imports, whereas commit 5fa327a0 removed node_mcf and retained only "golden_data/*.csv" under source_files. Additionally, the PR body contains empty []() links for EIA_Electricity and EIA_Petroleum test runs, and validation_config_EIA_*.json files still lack a scoped SQL_VALIDATOR date freshness check ([P2]).
Impact: Stale PR description and missing date freshness rules can mislead future maintainers and allow stale upstream series to pass validation undetected.
Recommendation: Update the PR title and body to reflect node_mcf removal, fill in the 2026_09_20 validation/GCS links, and add a scoped SQL_VALIDATOR date freshness rule in validation_config_EIA_*.json.
Added
node_mcf:"node_mcf": "gcs_output/<DATASET>/<DATASET>.mcf"to all 8 EIA imports inscripts/us_eia/opendata/manifest.json:EIA_Coal,EIA_Electricity,EIA_NaturalGas,EIA_NuclearOutages,EIA_Petroleum,EIA_International,EIA_SEDS,EIA_TotalEnergy.Regenerated Golden Data:
golden_observations_EIA_*.csv) .golden_summary_report_EIA_*.csv).VALIDATION: