Databricks Production Errors & Observability: Spark UI Triage & OOM Debugging

[!NOTE] Master data engineers are distinguished by their ability to open the Spark UI and instantly pinpoint memory bottlenecks, skewed partitions, and JVM garbage collection stalls.

1. Spark UI Triage Runbook: The 4 Key Tabs

1. Jobs & Stages Tab:
   • Inspect Task Event Timeline. If 199 tasks finish in 5 seconds while 1 task runs for 45 minutes, you have severe Data Skew.
2. Executors Tab:
   • Check Task Time (GC Time). If GC time exceeds 15% of total task time, executors are suffering memory pressure and swapping.
3. Storage Tab:
   • Check if cached DataFrames exceed executor storage memory and spill to disk.
4. SQL / DataFrame Tab:
   • View physical execution plan: identify SortMergeJoin vs BroadcastHashJoin and verify if Dynamic File Pruning (DFP) was applied.

Staff Triage: ConcurrentAppendException, SparkOutOfMemoryError, & Spill Diagnostics in Spark UI

• Resolving Delta ConcurrentAppendException / ProtocolChangedException: Happens when two jobs modify overlapping partitions/files without Row-Level Concurrency. Ensure delta.enableDeletionVectors = true, partition/cluster writes cleanly (MERGE ... ON target.date = source.date with explicit static partition predicates!), and serialize schema-evolution ALTER runs.
• Driver OOM vs Executor Spill Diagnosis: If the Driver dies before stages begin, check toPandas(), collect(), display() on millions of rows, or massive broadcast() hints (autoBroadcastJoinThreshold set too high). If Executors spill (Spill (Memory) vs Spill (Disk) in Spark UI Stage tab), inspect Skew (Max vs Median task duration) and Executor MemoryOverhead (especially for PySpark Arrow/Pandas UDFs!).
• Query Profile & Ganglia / Cluster Metrics: Use the Databricks Query Profile DAG (Photon metrics, rows output, peak memory) rather than guessing—if a single SortMergeJoin node reads 20x rows on one partition, fix the skew key (NULL / default "-1" join amplification!) immediately.