Skip to content

EIA Imports : Add node_mcf and regenerate golden data for EIA imports - #2224

Open
kartik-s21 wants to merge 3 commits into
datacommonsorg:masterfrom
kartik-s21:eia-opendata-node-mcf-goldens
Open

kartik-s21 wants to merge 3 commits into
datacommonsorg:masterfrom
kartik-s21:eia-opendata-node-mcf-goldens

Conversation

@kartik-s21

@kartik-s21 kartik-s21 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Added node_mcf:

  • Added "node_mcf": "gcs_output/<DATASET>/<DATASET>.mcf" to all 8 EIA imports in scripts/us_eia/opendata/manifest.json:
    • EIA_Coal, EIA_Electricity, EIA_NaturalGas, EIA_NuclearOutages, EIA_Petroleum, EIA_International, EIA_SEDS, EIA_TotalEnergy.

Regenerated Golden Data:

  • Re-generated observation goldens (golden_observations_EIA_*.csv) .
  • Re-generated summary report goldens (golden_summary_report_EIA_*.csv).
  • Validated that all 8 imports pass validation with 0 missing golden records.

VALIDATION:

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates several EIA golden data files by sorting the place identifiers alphabetically and reordering the columns in the summary reports. Additionally, it updates manifest.json to include the node_mcf path for multiple import configurations. There are no review comments, and I have no feedback to provide.

…enerate golden data

- Update node_mcf to wildcard pattern gcs_output/<DATASET>/*.mcf for all 8 imports to ingest both StatisticalVariable and StatVarGroup MCFs
- Add golden_data/*.csv to source_files in manifest.json
- Regenerate observation goldens matching top 100k places
- Sub-sample golden summary reports to concise representative records adhering to golden guidelines
@kartik-s21
kartik-s21 force-pushed the eia-opendata-node-mcf-goldens branch from 69d3257 to 22e6470 Compare September 17, 2026 04:36

@rohitkumarbhagat rohitkumarbhagat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the PR title, description, and validation configs before merging:

  1. PR Title & Summary: The current title (EIA Imports : Add node_mcf and regenerate golden data for EIA imports) and body (Added "node_mcf": "gcs_output/<DATASET>/<DATASET>.mcf" to all 8 EIA imports) still describe the reverted node_mcf addition rather than the latest commit (5fa327a0), which removes node_mcf and adds "golden_data/*.csv" to source_files.
  2. Validation Links: Several validation links in the PR description ([EIA_Electricity](), [EIA_Petroleum]()) are empty []() placeholders—please update them with the 2026_09_20 test run links.
  3. [P2] Date Freshness Validation (validation_config_EIA_*.json): Since golden_summary_report_EIA_*.csv excludes MaxDate and MaxDate varies across series in summary_report.csv, please add a scoped SQL_VALIDATOR date freshness check to validation_config_EIA_*.json.

"source_files": [
"gcs_output/COAL/COAL.txt"
"gcs_output/COAL/COAL.txt",
"golden_data/*.csv"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Documentation & Metadata Hygiene

Finding: The PR title and description still state that node_mcf was added to all 8 EIA imports, whereas commit 5fa327a0 removed node_mcf and retained only "golden_data/*.csv" under source_files. Additionally, the PR body contains empty []() links for EIA_Electricity and EIA_Petroleum test runs, and validation_config_EIA_*.json files still lack a scoped SQL_VALIDATOR date freshness check ([P2]).

Impact: Stale PR description and missing date freshness rules can mislead future maintainers and allow stale upstream series to pass validation undetected.

Recommendation: Update the PR title and body to reflect node_mcf removal, fill in the 2026_09_20 validation/GCS links, and add a scoped SQL_VALIDATOR date freshness rule in validation_config_EIA_*.json.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants