Loading Academy Hub...
Deploy enterprise Airflow at scale: Official Airflow Helm Chart on Kubernetes (EKS/GKE), Git-Sync sidecars, custom Docker images, and automated CI/CD DAG testing with `pytest`.
[!NOTE] Production Airflow clusters are deployed using the Official Apache Airflow Helm Chart on Kubernetes. DAGs are synchronized across pods in real-time using Git-Sync sidecar containers.
Before merging a pull request to main, run automated tests in your CI pipeline:
dag.test()).pytest Before Merge): Every PR touching dags/ must run an automated DagBag(include_examples=False) test asserting len(dag_bag.import_errors) == 0, valid owner/retries/sla tags, absence of top-level DB/network imports, and acyclic constraints before deploying via Git-Sync or Helm CI.Alternative Secrets Backends (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or HashiCorp Vault) directly in airflow.cfg ([secrets] backend) so Variables/Connections stay encrypted at rest outside Airflow Postgres and rotate automatically.KubernetesPodOperator or @task.virtualenv / @task.docker.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import pytest
from airflow.models import DagBag
@pytest.fixture(scope="session")
def dag_bag():
return DagBag(dag_folder="dags/", include_examples=False)
def test_no_import_errors(dag_bag):
"""Assert that all DAGs parse with zero syntax or import errors"""
assert len(dag_bag.import_errors) == 0, f"DAG import failures: {dag_bag.import_errors}"
def test_dag_cycles(dag_bag):
"""Assert that no DAG contains cyclical dependencies"""
for dag_id, dag in dag_bag.dags.items():
assert not dag.test_cycle(), f"Cyclic dependency detected in DAG: {dag_id}"git-sync: error cloning repository: git@github.com: Permission denied (publickey).