Conversation
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| Documentation | 5 minor |
| Security | 2 medium |
🟢 Metrics 5 complexity
Metric Results Complexity 5
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
There was a problem hiding this comment.
Code Review
This pull request introduces thread-safe lazy initialization of the BigQuery client using threading.local() in bq_executor.py. It also adds a concurrent deletion helper (delete_concurrent) in deleter.py using ThreadPoolExecutor to parallelize Spanner partitioned DML deletions, refactoring existing deletion methods to use it. Additionally, it updates linked_edge_generator.py to export data to both Edge and LinkedEdge tables, along with corresponding updates to tests. Feedback on the changes highlights two main areas of improvement: first, in linked_edge_generator.py, querying all typeOf edges without filtering can lead to a massive full-table scan and duplicate rows, so it is recommended to filter the query to only include subjects of containedInPlace edges and use DISTINCT to prevent primary key violations; second, in deleter.py, if delete_queries is empty, initializing ThreadPoolExecutor with max_workers=0 will raise a ValueError, so an early return guard should be added to handle empty query lists safely.
This PR updates the LinkedEdgeGenerator.run_linked_contained_in_place to also write to the LinkedEdge table
Also
Schema migration changes are in: datacommonsorg/datacommons#275 , but I double check on sequencing for submitting this
Mixer changes are currently flag gated (disabled everywhere)