Airflow Architecture: Webserver, Scheduler, Metadata DB & Executors

[!NOTE] Apache Airflow is the industry standard for programmatic workflow orchestration. Airflow does not process data itself; it orchestrates external compute engines (Databricks, Snowflake, BigQuery, Spark) through directed acyclic graphs (DAGs).

1. The Core Components of Airflow 2

1. Scheduler: Heart of Airflow. Monitors active DAGs, parses files, schedules tasks, and submits task instances to the executor queue. Supports multi-scheduler High Availability (HA) in Airflow 2+.
2. Metadata Database: Stores all state: task instance status, variables, connections, user permissions, and DAG definitions.
3. Executors:
   • CeleryExecutor: Fixed pool of long-running worker nodes listening to Redis/RabbitMQ queues. Fast task startup.
   • KubernetesExecutor: Launches an ephemeral Kubernetes pod for each task instance. Complete dependency isolation; scales to zero.
   • CeleryKubernetesExecutor: Hybrid model (Celery for lightweight short tasks, K8s for heavy compute).

Staff Architecture (2026 Airflow 3 Standard): Task Isolation, DAG Versioning & Executors

• Control Plane (Scheduler + Triggerer + API Server in Airflow 3): Airflow 3 cleanly separates metadata database access behind an internal Task Execution API (Edge/API Server) so remote workers no longer need direct Postgres credentials, adds native DAG Versioning, and supports event-driven Asset-Aware Scheduling.
• Choosing the Right Executor (Celery vs Kubernetes vs CeleryKubernetes): Use CeleryExecutor (with Autoscaling KEDA workers) for high-frequency lightweight tasks (<3s startup overhead); use KubernetesExecutor when tasks need strict per-task Docker image/dependency isolation or wildly varying GPU/RAM profiles (1 pod per task, ~15–30s pod spinup); use EdgeExecutor for remote on-prem/hybrid compute.
• Rule #1 of Airflow Workers: Airflow is an orchestrator, NEVER a compute engine. Never read multi-GB files or train ML models inside PythonOperator worker RAM—dispatch heavy work to Spark/EMR Serverless, Databricks, Snowflake, or BigQuery.