Sensors: S3KeySensor, Reschedule Mode & Complex Trigger Rules

[!NOTE] Sensors are specialized operators that wait for an external condition to become true (e.g., a file landing in S3, a table partition appearing, or a REST API endpoint returning 200).

1. Sensor Modes: poke vs reschedule (The Worker Starvation Trap)

• mode='poke' (Default): The sensor holds onto its worker slot for its entire duration, sleeping between check intervals. If a file takes 4 hours to arrive, that worker slot is locked for 4 hours doing nothing.
  • Risk: A few waiting sensors can exhaust all available worker slots, completely blocking all other production pipelines from running!
• mode='reschedule': The sensor checks the condition; if false, it releases the worker slot and reschedules itself in the metadata database for the next check interval.
  • Golden Rule: Always use mode='reschedule' for any sensor waiting longer than 2 minutes!

Staff Architecture: Deferrable Operators (deferrable=True), Deadlock Prevention & Pools

• Why Classic Sensors (poke Mode) Cause Worker Starvation: Standard polling sensors (poke mode) hold a full Worker process/pod hostage for hours sleeping between checks. Always set deferrable=True (e.g., S3KeySensor(..., deferrable=True), DatabricksRunNowOperator(..., deferrable=True), BigQueryInsertJobOperator(..., deferrable=True)) so the task yields immediately to the lightweight async Triggerer (asyncio), freeing 100% of Worker slots!
• Fallback When Custom Sensors Lack Deferrability: If a legacy provider lacks a Trigger class, strictly use mode="reschedule" rather than mode="poke" for waits exceeding 60 seconds so the worker slot returns to the pool between poke_interval checks.
• Pools (max_active_tis_per_dag) & exponential_backoff: Guard fragile downstream relational APIs or Snowflake staging tables using named Airflow Pools (slots=5) and retry_exponential_backoff=True + jitter so 100 failed tasks never DDoS your source ERP during recovery.