Local Docker Spark + Delta Lake + Apache Iceberg Cluster Setup Guide (Master, Worker, History Server & MinIO)

1. Why Every Serious Data Engineer Needs a Local JVM Spark Sandbox

In-browser WebAssembly runners (including DataArena’s Pyodide runner) are unbeatable for practicing DataFrame APIs, window expressions, SQL transformations, and daily interview drills anywhere with zero wait time. However, WebAssembly executes inside a single-threaded browser process — it does not launch real JVM executors, spill compressed shuffle blocks to NVMe, negotiate S3A multipart commits, or generate DAG timelines inside the Spark UI.

Spinning up an isolated multi-container cluster in Docker Desktop or Colima solves that gap completely for zero cloud spend. Within two minutes on a 16 GB laptop, you can trigger a realistic 100-to-1 skew join, watch executor cores stall inside Spark UI :4040, flip spark.sql.adaptive.skewJoin.enabled=true to watch AQE split the skewed partition, and inspect Parquet data files + _delta_log JSON transactions inside a local S3-compatible MinIO bucket.

2. Complete docker-compose.yml: Spark Master, Worker, History Server & MinIO

Below is a clean, production-patterned Docker Compose stack pairing Apache Spark 3.5, a shared event-log volume for the History Server on port 18080, and MinIO acting as local S3 object storage on ports 9000 (API) and 9001 (Web Console). Notice we mount local ./jobs and ./spark-events folders so your Python scripts and event logs survive container restarts cleanly.

services:
  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password123
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - ./minio-data:/data

  spark-master:
    image: apache/spark:3.5.3
    command: /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master
    environment:
      SPARK_NO_DAEMONIZE: "true"
    ports:
      - "8080:8080"
      - "7077:7077"
      - "4040:4040"
    volumes:
      - ./jobs:/opt/spark/jobs
      - ./spark-events:/tmp/spark-events

  spark-worker:
    image: apache/spark:3.5.3
    depends_on: [spark-master]
    command: /opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
    environment:
      SPARK_NO_DAEMONIZE: "true"
      SPARK_WORKER_CORES: "4"
      SPARK_WORKER_MEMORY: "4g"
    volumes:
      - ./jobs:/opt/spark/jobs
      - ./spark-events:/tmp/spark-events

  spark-history:
    image: apache/spark:3.5.3
    depends_on: [spark-master]
    command: /opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer
    environment:
      SPARK_NO_DAEMONIZE: "true"
      SPARK_HISTORY_OPTS: "-Dspark.history.fs.logDirectory=/tmp/spark-events"
    ports:
      - "18080:18080"
    volumes:
      - ./spark-events:/tmp/spark-events

3. Running spark-submit With Delta Lake, Iceberg & AQE Skew Verification

Once your containers are healthy (docker compose up -d), download or attach the Maven coordinates for Delta Lake (io.delta:delta-spark_2.12:3.2.0), Apache Iceberg (org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1), and AWS Hadoop S3A (org.apache.hadoop:hadoop-aws:3.3.4) directly via --packages on spark-submit. Enabling spark.eventLog.enabled=true guarantees every job writes full task timelines into /tmp/spark-events so you can audit completed jobs at http://localhost:18080.

docker compose exec spark-master /opt/spark/bin/spark-submit \
  --master spark://spark-master:7077 \
  --packages io.delta:delta-spark_2.12:3.2.0,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,org.apache.hadoop:hadoop-aws:3.3.4 \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
  --conf spark.sql.adaptive.enabled=true \
  --conf spark.sql.adaptive.skewJoin.enabled=true \
  --conf spark.eventLog.enabled=true \
  --conf spark.eventLog.dir=/tmp/spark-events \
  --conf spark.hadoop.fs.s3a.endpoint=http://minio:9000 \
  --conf spark.hadoop.fs.s3a.access.key=admin \
  --conf spark.hadoop.fs.s3a.secret.key=password123 \
  --conf spark.hadoop.fs.s3a.path.style.access=true \
  /opt/spark/jobs/aqe_skew_lab.py

4. What to Inspect in Spark UI (:4040 / :18080) During Self-Practice

Never stop at seeing "Job finished". Open the SQL / DataFrame tab inside Spark History Server at localhost:18080 and inspect four specific boxes inside the physical DAG: Exchange boxes (each represents an expensive network shuffle boundary), AdaptiveSparkPlan isFinalPlan=true (click to see where initial plans mutated into CustomShuffleReader skew splits or BroadcastHashJoin downgrades), Files Read & Partition Filters Pushed (confirm pruning skipped unused directories), and Stages -> Task Summary Metrics (verify GC Time stays below 10% of Task Time and Spill Disk stays near zero).

5. Hands-On Experiment: Intentional Skew Script & Reading the Recovery Plan

Do not stop at launching containers — intentionally manufacture a skewed join locally so you recognize AQE CustomShuffleReader plans immediately in live interviews. Save the Python script below inside ./jobs/aqe_skew_lab.py, run it once with spark.sql.adaptive.skewJoin.enabled=false (noting the stalled straggler task inside :18080), then flip it back to true and inspect how Spark splits partition 0 into balanced sub-partitions written straight to MinIO Delta storage.

Once your script completes, open the MinIO Console on port 9001 and inspect the actual Parquet chunks alongside `_delta_log/00000000000000000000.json`. Seeing the physical transaction log entries (`add`, `metaData`, `commitInfo`) alongside byte-level column statistics makes optimistic concurrency control and Parquet data-skipping click permanently — turning abstract whiteboard buzzwords into real operational fluency during senior system design interviews.

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("AQE-Skew-Lab").getOrCreate()

# Create an intentionally skewed 2M-row dataset where 85% of traffic hits tenant 999
skewed_orders = spark.range(0, 2_000_000).withColumn(
    "tenant_id", F.when((F.col("id") % 10) < 8, F.lit(999)).otherwise(F.col("id") % 500)
).withColumn("amount", (F.col("id") % 250).cast("double"))

# Force shuffle join by disabling broadcast threshold temporarily so you observe skew in Spark UI
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
tenants = spark.range(0, 1000).withColumnRenamed("id", "tenant_id").withColumn("tier", F.lit("ENTERPRISE"))

joined = skewed_orders.join(tenants, on="tenant_id", how="inner")
joined.groupBy("tier").agg(F.round(F.sum("amount"), 2).alias("total_rev")).show()

# Persist Parquet / Delta straight to your local MinIO bucket (create bucket 'lakehouse' in :9001 first)
joined.limit(50000).write.format("delta").mode("overwrite").save("s3a://lakehouse/silver/skew_test")
print("Inspect DAG & Task skew metrics at http://localhost:18080")

6. Troubleshooting Common Local Docker Spark & S3A Commit Issues

Three predictable configuration hiccups trip up engineers during initial Docker Spark setups: First, `ClassNotFoundException: org.apache.hadoop.fs.s3a.S3AFileSystem` happens whenever Maven `--packages` omits `hadoop-aws` matching your image’s Hadoop 3.3.x binary version cleanly — always align Hadoop versions across `spark-submit`. Second, `UnknownHostException: minio` inside local IDE tests occurs when running PySpark outside the Docker Compose bridge network (`http://localhost:9000` from host Python vs `http://minio:9000` inside worker containers). Third, forgetting `spark.hadoop.fs.s3a.path.style.access=true` forces AWS virtual-hosted bucket DNS lookups (`lakehouse.minio:9000`) that fail immediately against local MinIO.

Keep a clean `.env.local` and Makefile target (`make up`, `make submit`, `make logs`, `make down`) wrapping your stack cleanly so spinning up multi-container Iceberg REST catalogs or Kafka streams for weekend experimentation takes ten seconds instead of an afternoon of environment debugging.