Loading Academy Hub...
Orchestrate enterprise GCP pipelines: Google Cloud Composer (Airflow 2), BigQueryInsertJobOperator, dynamic DAG generation, Workload Identity, and cross-project permissions.
[!NOTE] Google Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. In GCP, Composer serves as the control plane orchestrating Cloud Dataflow, Dataproc, and BigQuery execution.
In modern Airflow 2 on GCP, legacy operators (BigQueryExecuteQueryOperator) are superseded by BigQueryInsertJobOperator:
BigQueryInsertJobOperator, DataprocCreateBatchOperator, DataflowStartFlexTemplateOperator)—never load 5GB Pandas DataFrames inside a Composer PythonOperator, which causes SIGKILL Pod evictions and scheduler starvation.30s during DagBag parsing!) and leverage Workload Identity rather than exporting JSON service-account key files into Airflow Connections.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'data-engineering',
'retries': 2,
'retry_delay': timedelta(minutes=5),
}
with DAG(
dag_id='prod_financial_reporting_dag',
default_args=default_args,
schedule_interval='0 2 * * *',
start_date=datetime(2026, 1, 1),
catchup=False
) as dag:
run_daily_fact_aggregation = BigQueryInsertJobOperator(
task_id='aggregate_daily_facts',
configuration={
"query": {
"query": """
MERGE `enterprise-lakehouse-prod.finance.fact_daily_summary` t
USING `enterprise-lakehouse-prod.finance.stg_daily_transactions` s
ON t.account_id = s.account_id AND t.report_date = s.report_date
WHEN MATCHED THEN UPDATE SET t.total_amount = s.total_amount
WHEN NOT MATCHED THEN INSERT ROW;
""",
"useLegacySql": False,
}
}
)airflow.exceptions.AirflowTaskTimeout: Task aggregate_daily_facts exceeded timeout of 3600 seconds.