Data Engineering Production Error & Troubleshooting Encyclopedia
Select any incident or category in the left panel to inspect full stack traces, telemetry log locations, verified production fixes, alerting SLOs, and interview takeaways.
Apache Spark / Databricks / EMRCritical Severity
Target MTTR: 15–30 min
Executor OOM: Container killed by YARN / K8s for exceeding memory limits (Exit Code 137)
Production Stack Trace Signature
ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 16.4 GB of 16.0 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. Exit code 137.
Root Cause Breakdown
Off-heap memory exhaustion or heavy Python worker memory usage. PySpark passes data via Py4J/Arrow IPC pipes to Python daemon processes. High cardinality UDFs, wide row serialization, or excessive off-heap buffers exceed `spark.executor.memoryOverhead` (defaults to only 10% of executor memory).
1. Increase Memory Overhead in spark-defaults.conf or spark-submit:
--conf spark.executor.memoryOverhead=4096m (or 25% of executor memory for heavy Python ETL)
2. If using Python UDFs, replace with Native PySpark SQL functions or Vectorized Pandas UDFs (@pandas_udf).
3. Check for high-cardinality groupBy or skewed shuffle partitions; increase `spark.sql.shuffle.partitions` from default 200 to 1200+.
4. Ensure JVM Garbage Collection is tuned: `spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35`.
Data Engineering Production Error & Troubleshooting Encyclopedia
When a 2:00 AM batch pipeline pages the on-call data engineer, reading a 60-line Java or cloud stack trace calmly under pressure separates senior engineers from beginners. Most outages trace back to a small family of failure modes: off-heap memory overcommit (Exit 137), driver collect OOMs, optimistic concurrency collisions on Delta/Iceberg tables, or object-store throttling.
This Production Error Encyclopedia documents 18 high-severity incident runbooks across PySpark, Databricks Delta Lake, Snowflake, Apache Airflow, AWS Athena/Glue/S3, Azure Synapse/ADF, and relational SQL MERGE pipelines—showing exact error signatures, where to inspect logs, verified production fixes, prevention alerts, and what hiring panels expect you to explain.
Complete Index: All 18 Production Incident Runbooks
Apache Spark / Databricks / EMRMTTR: 15–30 min
Executor OOM: Container killed by YARN / K8s for exceeding memory limits (Exit Code 137)
Off-heap memory exhaustion or heavy Python worker memory usage. PySpark passes data via Py4J/Arrow IPC pipes to Python daemon processes. High cardinality UDFs, wide row serialization, or excessive off-heap buffers exceed `spark.executor.memoryOverhead` (defaults to only 10% of executor memory).
Apache Spark / Databricks / GlueMTTR: 15–30 min
FetchFailedException & Shuffle Block Not Found
The executor hosting the shuffle block died unexpectedly (often from an OutOfMemory or Hardware Spot Instance termination) while the downstream stage was attempting to fetch intermediate shuffle files over Netty.
Apache Spark / LakehouseMTTR: 15–30 min
Data Skew / 199 Tasks Complete in 2 Minutes, Task 200 Hangs for 3 Hours
A join key or groupBy key has severe distribution imbalance (e.g. `NULL`, `UNKNOWN`, or default ID `0` accounts for 70% of rows). All matching rows hash to a single partition on a single executor core.
Snowflake Cloud Data WarehouseMTTR: 15–30 min
Snowflake Query Spilling to Remote Storage (Severe Performance Degradation)
The active Virtual Warehouse has insufficient RAM for the join or aggregate memory hash table. When local SSD cache fills up, Snowflake spills uncompressed intermediate buffers to slow S3/Blob storage, causing a 20x latency penalty.
Snowflake SnowpipeMTTR: 15–30 min
Snowpipe Ingestion Silent Failure / Files Missing from Destination Table
Source file schema drift or invalid data type (e.g. an API upstream started sending string "N/A" into an integer column). Default Snowpipe `ON_ERROR = ABORT_STATEMENT` causes silent file drop without writing to target.
Databricks / Delta LakeMTTR: 15–30 min
ConcurrentModificationException: Conflict with concurrent commit in Delta transaction log
Two parallel pipeline runs or write jobs attempted to commit changes to the exact same partition of a Delta table at the same timestamp without optimistic concurrency coordination.
Apache AirflowMTTR: 15–30 min
Zombie Task / TaskInstance killed because worker stopped heartbeat
The Airflow worker process executing the task was terminated abruptly by the Operating System (typically Linux Out-Of-Memory Killer `SIGKILL 9` because a Python task loaded too much data into local memory), or a long synchronous query blocked the Python thread from sending heartbeats to Postgres.
Network firewall rules, expired Service Principal / Managed Identity secrets, or transient connection drops on high-concurrency connections to Azure Synapse / SQL DB.
PostgreSQL / MySQL / Enterprise OLTPMTTR: 15–30 min
Database Transaction Deadlock (SQLSTATE 40P01)
Two concurrent ETL transactions are updating multiple tables or rows in reverse order. Transaction A locks Table 1 then requests Table 2; Transaction B locks Table 2 then requests Table 1.
Databricks Delta Lake / SparkMTTR: 15–30 min
Delta ConcurrentAppendException & Metadata Conflict Under Parallel Writers
Two concurrent writers (e.g. overlapping country backfill tasks or a streaming writer racing with OPTIMIZE/unscoped MERGE) read the same snapshot version and wrote conflicting commits without disjoint partition predicates.
Spark Structured Streaming / KafkaMTTR: 15–30 min
Structured Streaming State Store OOM & Infinite Checkpoint Growth
Calling dropDuplicates(["id"]) or groupBy() aggregation WITHOUT withWatermark(), or omitting the watermark column from dropDuplicates(["id", "event_time"]), forces the State Store to cache every key forever until executors run out of RAM.
Apache Iceberg / Athena / Trino / S3MTTR: 15–30 min
Streaming writers committing every 30 seconds created 15,000 tiny manifest files and thousands of historical snapshots. Query planners must read thousands of small S3 Avro manifests just to plan one SELECT query.
Airflow Metadata DB Connection Exhaustion & Sensor Worker Slot Deadlock
Hundreds of parallel DAG runs or long-waiting `FileSensor`/`ExternalTaskSensor` instances defaulting to `mode="poke"` hold live worker processes and open Postgres sessions continuously while waiting for upstream files, starving downstream transform tasks and exhausting Postgres `max_connections`.
Apache Spark / EMR / DatabricksMTTR: 15–25 min
BroadcastTimeoutException: Could not execute broadcast exchange in 300 secs
Spark chose a BroadcastHashJoin for a table whose Catalyst metadata size looked < 10MB, OR the dimension pipeline feeding the broadcast is doing an expensive upstream shuffle/scan that takes > 300s before the Driver can collect and ship blocks to executors.
Default `MAX_CONCURRENCY_LEVEL = 8` on a single-cluster Virtual Warehouse forced dozens of dbt/BI queries to queue sequentially, OR a full-warehouse BI refresh blocked shorter SLA ETL transforms on the same warehouse.
Azure Synapse Dedicated SQL PoolMTTR: 20–40 min
Azure Synapse Resource Class Queuing, TEMPDB Full & Distribution Skew
All queries ran under `largerc` or `xlargerc` consuming 25–70% of DWU concurrency slots per query, OR a large fact table used `HASH(nullable_or_skewed_col)` packing 80% of rows onto 2 out of 60 compute distributions.
Snowflake / BigQuery / Delta / Postgres SQLMTTR: 10–15 min
Nondeterministic MERGE Failure: Multiple Source Rows Matched Same Target Row
Upstream CDC / Kafka / API ingestion delivered duplicate events sharing the same `unique_key` inside the same incremental batch, violating SQL `MERGE` deterministic 1-to-1 matching semantics.
Top 10 Frequently Asked Questions (Engineers & Hiring Panels)
1. What causes Spark Container Killed by YARN / Kubernetes with Exit Code 137?
Exit code 137 (128 + SIGKILL 9) means the OS kernel or cluster manager killed the executor container because total RSS memory—JVM heap plus off-heap Python worker, PyArrow buffers, or Netty shuffle overhead—exceeded the pod memory limit. Fix it by increasing spark.executor.memoryOverhead, lowering spark.sql.shuffle.partitions skew, or switching row UDFs to native Spark SQL expressions.
2. How do you fix Driver OutOfMemoryError vs Executor OutOfMemoryError?
Driver OOM happens when df.collect(), df.toPandas(), or huge broadcast variables pull gigabytes back to the single driver JVM. Executor OOM happens during wide shuffles, skewed groupBy aggregations, or exploding nested arrays on worker nodes. Always replace .collect() with .limit(n) or write directly to cloud storage.
3. Why does BroadcastTimeoutException (300s) happen even on a small dimension table?
If the "small" table is actually the output of an expensive upstream join or unpartitioned CSV read, computing it before broadcasting can exceed the 300-second spark.sql.broadcastTimeout. Either materialize/cache the filtered dimension first, raise broadcastTimeout, or disable broadcast hints on dynamic pipelines.
4. How do you resolve Delta Lake ConcurrentAppendException or ConcurrentDeleteReadException?
Delta Lake uses Optimistic Concurrency Control (OCC). Two concurrent MERGE or OPTIMIZE jobs updating the same partition conflict on commit. Fix it by partitioning writes on disjoint keys (e.g., date + region) or enabling isolation levels and automated job-level retry with exponential backoff.
5. What triggers HIVE_PARTITION_SCHEMA_MISMATCH or Parquet schema drift errors in AWS Athena & Glue?
Upstream producers changing an integer column to bigint/double or adding nested JSON fields makes older Parquet footers incompatible with the Glue Data Catalog table schema. Enable mergeSchema cautiously on curated Silver loads, enforce explicit schema contracts at Bronze ingestion, and cast columns before appending.
6. Why do Apache Airflow tasks get marked as Zombie tasks?
A task turns into a Zombie when its worker process crashes silently (OOMKilled or spot instance preemption) or loses database heartbeat connectivity for longer than scheduler_zombie_task_threshold. Give heavy Spark/Python operators dedicated memory requests, offload compute via KubernetesPodOperator or DatabricksSubmitRunOperator, and set idempotent retries.
7. What does Snowflake "SQL compilation error: Statement reached its memory/time limit" indicate?
Complex deeply nested CTEs, thousands of OR predicates, or giant IN lists blow up query optimizer compilation before execution even starts. Break monolithic SQL into materialized transient staging tables or replace 5,000-item IN (...) clauses with a JOIN against a temporary lookup table.
8. Why do AWS S3 pipelines fail with 503 SlowDown ("Please reduce your request rate")?
Bursting thousands of concurrent Spark tasks listing or writing files under the same narrow S3 prefix triggers S3 per-prefix request limits (3,500 PUT / 5,500 GET req/sec per prefix). Reduce output task parallelism via coalesce/AQE, avoid unnecessary recursive listings, and use date/hash prefixes.
9. How do you diagnose FetchFailedException / ShuffleMapStage failures in Spark?
FetchFailedException means a reducer could not fetch shuffle blocks from an upstream executor—usually because that upstream node died from spot preemption, disk full on local /tmp shuffle directories, or OOM. Enable External Shuffle Service / push-based shuffle or use EBS autoscaling on shuffle-heavy EMR clusters.
10. What is the fastest way to structure an incident debugging answer in an engineering interview?
Follow a 4-step SRE structure: (1) isolate the failing stage/log layer (driver vs executor vs catalog vs storage), (2) confirm the root cause metric (skew ratio, RSS ceiling, or lock contention), (3) apply an immediate safe mitigation (backfill/restart/memory bump), and (4) add a permanent guard rail (schema validation, DLQ, or partition alert).