Invoke airflow dag from import helper

[vish-cs]

Add import_automation_airflow option in import-helper to trigger Cloud Composer DAGs

Summary

Adds support for triggering Cloud Composer (Airflow) DAGs from import-helper when post_process=import_automation_airflow is specified in CDA transfer completion events (/imports/feed), while preserving existing Cloud Workflows execution for spanner_ingestion_workflow and import_automation_workflow.

Key Changes

• Airflow DAG Invocation (utils/imports.py): Added invoke_import_automation_airflow to trigger DAG runs via the Composer REST API (/api/v1/dags/{dag_id}/dagRuns), supporting both IAP OIDC ID tokens (AIRFLOW_IAP_CLIENT_ID) and default cloud-platform OAuth2 credentials.
• Event Routing (routes/events.py):
  • Added post_process == 'import_automation_airflow' branch in handle_feed_event.
  • Uses the dag_id Pub/Sub attribute when provided, defaulting (and falling back on HTTP 404) to the generic import_automation_workflow DAG.
• Deployment Config (config.py, deploy-services.yaml, cloudbuild.yaml): Added AIRFLOW_WEB_SERVER_URL, AIRFLOW_DEFAULT_DAG_ID, and AIRFLOW_IAP_CLIENT_ID configuration and Cloud Build substitutions.
• Tests (app_test.py): Added unit tests for import_automation_airflow event handling, generic DAG routing, 404 fallback, and authentication modes.

TAG=agy
CONV=e8d6e7b3-338a-4bed-8f9f-3e7c9ca97a52

[codacy-production]

Not up to standards ⛔

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 14 complexity

Metric	Results
Complexity	14

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[gemini-code-assist]

Code Review

This pull request integrates Cloud Composer (Airflow) into the import automation workflow, allowing the import-helper service to trigger Airflow DAGs via REST API calls, complete with fallback logic and unit tests. The review feedback points out critical improvements: avoiding a globally hardcoded _AIRFLOW_WEB_SERVER_URL in cloudbuild.yaml to prevent environment cross-talk, using an OIDC ID token instead of an OAuth 2.0 Access Token to successfully authenticate through Identity-Aware Proxy (IAP) on Cloud Composer v2, and removing the hardcoded default tenant URL in config.py to prevent environment-specific leaks.