Loading Academy Hub...
Scale Airflow workflows dynamically: Dynamic Task Mapping (`.expand()`), Task Concurrency Pools, priority weights, and SLA missed callbacks.
[!NOTE] In legacy Airflow, iterating over an unknown number of files or tables required re-generating the entire DAG file. Airflow 2 introduced Dynamic Task Mapping (
.expand()), creating task instances at runtime based on the output of upstream tasks.
.expand()Upstream task returns a list of items (e.g., list of S3 prefixes discovered at runtime). The downstream task is invoked with .expand(), automatically spawning parallel task instances in the Airflow UI without changing the DAG code.
.expand()), Setup/Teardown & Map Limits.expand()) vs Parse-Time Loops: Traditional for f in s3_list: loops at DAG top-level fail whenever file counts fluctuate at runtime and crush the Scheduler. process_file.expand(file_path=list_s3_files()) determines mapped task fanout cleanly at runtime (map_index=0..N) and tracks retries/logs per individual shard!max_map_length Guardrail (1,024 Default): If an upstream task discovers 15,000 tiny S3 keys, blindly expanding 15,000 mapped TaskInstances will overwhelm Airflow’s Postgres Scheduler (max_map_length). Instead, batch keys into groups of 100 files (chunk_files() -> List[List[str]]) so .expand() spawns a clean 150 worker tasks..as_setup() / .as_teardown()): Attach temporary EMR/Dataproc cluster creation and guaranteed destruction using create_cluster >> work >> delete_cluster.as_teardown(setups=create_cluster) so cloud clusters still self-destruct even when intermediate transform tasks fail!Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
from airflow.decorators import dag, task
from datetime import datetime
@dag(dag_id='dynamic_parallel_ingest', schedule='@daily', start_date=datetime(2026, 1, 1), catchup=False)
def dynamic_ingest_dag():
@task
def discover_landing_files() -> list:
# Dynamically returns list of files discovered in S3
return ["s3://lakehouse/drop/orders.csv", "s3://lakehouse/drop/users.csv", "s3://lakehouse/drop/txns.csv"]
@task(pool='lakehouse_ingest_pool', max_active_tis_per_dag=5)
def process_file(s3_path: str):
print(f"Processing file: {s3_path}")
return f"Processed {s3_path}"
# Dynamic mapping: spawns 3 parallel task instances at runtime
files = discover_landing_files()
process_file.expand(s3_path=files)
dynamic_ingest_dag()OperationalError: (psycopg2.OperationalError) FATAL: remaining connection slots are reserved for non-replication superuser connections.