Airflow Optimization Playbook: Scheduler Heartbeats & Top-Level Code Traps

[!NOTE] Slow Airflow task execution is almost always caused by Scheduler contention resulting from heavy top-level code in DAG files.

1. Top Performance Tuning Levers

1. Eliminate Top-Level Code:
   • The scheduler executes the global scope of every .py file in dags/ every few seconds.
   • Any requests.get(), boto3 connection, or database query in global scope stalls the scheduler parsing loop.
2. Tuning min_file_process_interval:
   • Increase from default 30s to 120s or 300s in production to give the scheduler more CPU time for task queuing.
3. Use .airflowignore:
   • Ignore test files, documentation, and utility libraries so the scheduler spends zero cycles parsing non-DAG scripts.

Staff Optimization: Killing Top-Level DAG Parsing Traps & Scheduler CPU Bottlenecks

• The #1 Airflow Production Anti-Pattern (Top-Level Network/DB Calls): The Airflow Scheduler re-parses every .py file inside dags/ every 30 seconds (min_file_process_interval). Putting Variable.get(), boto3.client("s3").list_objects(), or PostgresHook().get_records() outside a task body executes that API call thousands of times per hour—crashing Metadata DBs and stalling DAG scheduling! Always place Variable.get(), Hooks, and heavy imports (pandas, torch, pyspark) strictly inside execute() / @task bodies (or use Jinja {{ var.value.my_key }}).
• Dynamic DAG Generation Scaling: When generating 200 table ingestion DAGs from YAML/JSON configs, load small static YAML files cleanly rather than querying live databases at parse time, or consolidate similar tables into parameterized Dynamic Task Mapping runs.
• Metadata DB Retention Hygiene: Schedule weekly airflow db clean (--clean-before-timestamp)—unpruned task_instance, xcom, and log tables across millions of historical runs silently slow every Scheduler query to a crawl.