Local Docker Spark + Delta / Iceberg Cluster Lab
DataArena's interactive Problems and Cloud Labs run on a fast in-browser Pyodide WebAssembly sandbox so you can drill DataFrame and SQL logic anywhere with zero setup. When you want to inspect real JVM Catalyst physical plans, AQE skew splitting, shuffle spills, and Spark UI (:4040 / :18080), spin up this 3-container Docker Compose cluster on your laptop in 60 seconds.
docker compose up -d && docker exec dataarena-spark-master \ /opt/spark/bin/spark-submit --master spark://spark-master:7077 \ /opt/spark/work-dir/jobs/skew_aqe_demo.py
Works on Windows (WSL2 / Docker Desktop), macOS, and Linux.
| Capability | Browser Pyodide Sandbox (Here) | Local Docker JVM Spark (Below) |
|---|---|---|
| PySpark DataFrame & SQL API | Instant in-browser testing across 243 problems | Full Apache Spark 3.5.3 JVM + Catalyst runtime |
| Spark UI (:4040) & History (:18080) | Simulated architecture & lineage only | Live Stage DAGs, Task Skew percentiles, GC & Spill bytes |
| AQE, Broadcast & Iceberg / Delta JARs | Lightweight Python shim (no JVM JARs) | Real `--packages` Maven resolution (`delta-spark`, `iceberg-spark-runtime`) |
Spark Master + Worker + History Server with shared event logs
services:
spark-master:
image: apache/spark:3.5.3-python3
container_name: dataarena-spark-master
command: /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master
environment:
SPARK_MASTER_HOST: spark-master
SPARK_MASTER_PORT: 7077
SPARK_MASTER_WEBUI_PORT: 8080
ports:
- "8080:8080" # Spark Master Web UI
- "7077:7077" # Spark Master RPC
- "4040:4040" # Live Application Spark UI (Jobs / Stages / SQL DAG)
volumes:
- ./jobs:/opt/spark/work-dir/jobs
- ./warehouse:/opt/spark/work-dir/warehouse
- ./spark-events:/tmp/spark-events
spark-worker:
image: apache/spark:3.5.3-python3
container_name: dataarena-spark-worker
depends_on:
- spark-master
command: /opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
environment:
SPARK_WORKER_CORES: 2
SPARK_WORKER_MEMORY: 2g
volumes:
- ./jobs:/opt/spark/work-dir/jobs
- ./warehouse:/opt/spark/work-dir/warehouse
- ./spark-events:/tmp/spark-events
spark-history:
image: apache/spark:3.5.3-python3
container_name: dataarena-spark-history
command: /opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer
environment:
SPARK_HISTORY_OPTS: "-Dspark.history.fs.logDirectory=/tmp/spark-events"
ports:
- "18080:18080" # Spark History Server UI
volumes:
- ./spark-events:/tmp/spark-events
1.5M skewed-join workload to observe AQE skew splitting in Spark UI
# Save as ./jobs/skew_aqe_demo.py
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
spark = (
SparkSession.builder
.appName("DataArena-Skew-AQE-Lab")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.adaptive.skewJoin.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
.config("spark.eventLog.enabled", "true")
.config("spark.eventLog.dir", "/tmp/spark-events")
.getOrCreate()
)
# 1. Generate skewed 1.5M-row fact table (80% traffic hits merchant_id = 1)
orders = (
spark.range(0, 1_500_000)
.withColumn(
"merchant_id",
F.when(F.col("id") % 10 < 8, F.lit(1)).otherwise((F.col("id") % 500) + 2)
)
.withColumn("amount", (F.col("id") % 250) + 1.0)
)
# 2. Dimension lookup table
merchants = (
spark.range(1, 502)
.withColumnRenamed("id", "merchant_id")
.withColumn("tier", F.when(F.col("merchant_id") == 1, F.lit("MEGA")).otherwise(F.lit("SMB")))
)
# Force SortMergeJoin (bypass broadcast threshold) to watch AQE split skewed partition!
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
joined = (
orders.join(merchants, on="merchant_id", how="inner")
.groupBy("tier")
.agg(F.count("*").alias("tx_count"), F.round(F.sum("amount"), 2).alias("gmv"))
)
print("=== Catalyst Physical Plan ===")
joined.explain(mode="formatted")
joined.show()
spark.stop()
mkdir -p spark-local/jobs spark-local/warehouse spark-local/spark-events && cd spark-local # Place docker-compose.yml in spark-local/ and skew_aqe_demo.py in spark-local/jobs/ docker compose up -d
docker exec -it dataarena-spark-master /opt/spark/bin/spark-submit \ --master spark://spark-master:7077 \ --packages io.delta:delta-spark_2.12:3.2.0 \ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \ /opt/spark/work-dir/jobs/skew_aqe_demo.py
docker exec -it dataarena-spark-master /opt/spark/bin/spark-submit \ --master spark://spark-master:7077 \ --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1 \ --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \ --conf spark.sql.catalog.local.type=hadoop \ --conf spark.sql.catalog.local.warehouse=/opt/spark/work-dir/warehouse \ /opt/spark/work-dir/jobs/skew_aqe_demo.py
docker compose logs -f spark-worker docker compose down -v
What to Inspect in Your Local Spark UIs
Save this setup in a public GitHub repo alongside one custom Medallion pipeline (CSV/JSON -> Bronze -> Deduped Silver Delta/Iceberg -> Gold Mart) plus screenshots of your Spark History Server DAG before and after AQE tuning. Next, pair it with our Performance Optimization Bible and 6-Round Interview Arena.