USFEMA_FloodInsuranceClaims: Optimize download and transformation pipeline - #2217
Open
kartik-s21 wants to merge 2 commits into
Open
kartik-s21 wants to merge 2 commits into
kartik-s21 wants to merge 2 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request re-engineers the FEMA flood insurance claims import pipeline, replacing a slow, sequential, row-by-row legacy processor with a highly optimized, vectorized, multi-process chunked pipeline. It introduces direct bulk CSV downloading with paginated API fallback, updates the manifest and documentation, and adds comprehensive unit tests. Feedback suggests further optimizing the generation of StatVar names in process.py by replacing .apply() with .map() using pre-computed unique flood zone names to avoid Python function call overhead when generating millions of observations.
kartik-s21
force-pushed
the
usfema-flood-claims-optimization
branch
2 times, most recently
from
September 10, 2026 10:15
4d7f6d6 to
56746b1
Compare
kartik-s21
force-pushed
the
usfema-flood-claims-optimization
branch
4 times, most recently
from
September 11, 2026 10:57
58c6d1a to
3848d33
Compare
kartik-s21
force-pushed
the
usfema-flood-claims-optimization
branch
from
September 11, 2026 12:15
c6a8334 to
7bb1d11
Compare
…ipeline - Re-engineer download pipeline with direct bulk CSV streaming and API pagination fallback. - Replace sequential row iteration in process.py with multi-process vectorized pipeline. - Use concurrent.futures.ProcessPoolExecutor with spawn context for safe multiprocessing. - Add FIPS zero-padding for tract and county places to prevent data loss for states 01-09. - Enforce atomic publishing for observations, MCF, TMCF, and counters. - Improve error handling with non-zero exit on empty aggregations. - Replace eval() with ast.literal_eval() for secure mapping parsing. - Update manifest.json node_mcf pattern and source_files. - Expand README documentation and add comprehensive unit test suite.
kartik-s21
force-pushed
the
usfema-flood-claims-optimization
branch
from
September 11, 2026 12:38
7bb1d11 to
0c411ef
Compare
balit-raibot
self-requested a review
September 17, 2026 03:19
balit-raibot
approved these changes
Sep 17, 2026
- Fix swallowed IOError and stage chunks before atomic move in fema_download.py. - Enforce record count verification and raise on incomplete pagination. - Add --output_dir flag and resolve paths relative to script directory. - Format CountOfClaims as clean integer strings and lower chunk size floor to 25k in process.py. - Remove dead --config_file flag and clean pv_map argument in manifest.json and process.py. - Remove unused retry dependency from README prerequisites. - Add unit tests for download fallback paths and IOError handling in fema_download_test.py. - Regenerate test_data/flood_insurance_claims_output.csv with current pipeline output.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| mock_get_total_records.return_value = 2 | ||
|
|
||
| def side_effect(url, output_folder, **kwargs): | ||
| if "fake-bulk.com" in url: |
| mock_get_total_records.return_value = 2 | ||
|
|
||
| def side_effect(url, output_folder, **kwargs): | ||
| if "fake-bulk.com" in url: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updated the download and data processing pipeline for
USFEMA_FloodInsuranceClaimsto eliminate multi-hour runtime bottlenecks and improve error resilience.fema_download.py)shutil.moveonce the full dataset is downloaded, preventing data duplication on retries.IOErrorto fatal exceptions to prevent downstream processing of truncated files. Added verification assertingrecords_downloaded >= total_records.--output_dirand resolved all paths relative to the script directory to prevent root-directory path coupling.process.py)ProcessPoolExecutorwithmultiprocessing.get_context('spawn').25,000rows (max(25000, 2750000 // (workers * 2))), ensuring full core saturation for provisioned 64-core Cloud Batch instances.CountOfClaimsobservations as clean integer strings (1,2) while formatting monetary metrics to 2 decimal places. Preserves all-NaN groups usingmin_count=1..0float artifacts to prevent data loss in states with FIPS < 10.--config_fileflag and cleaned--pv_maparguments.manifest.json)counters/*insource_filesfor pipeline monitoring.process.py.fema_download_test.py,process_test.py,test_data/)test_data/flood_insurance_claims_output.csv(6,929 rows) to reflect bothP1MandP1Yresolutions.retrypackage fromREADME.mdprerequisites.process_test.py,fema_download_test.py):