Free Forever · Local Cluster Kit
Spark 3.5.3 + Delta / Iceberg
Windows / macOS / Linux (Docker Desktop)
Browser Pyodide Sandbox vs. Real JVM Spark Cluster

Local Docker Spark + Delta / Iceberg Cluster Lab

DataArena's interactive Problems and Cloud Labs run on a fast in-browser Pyodide WebAssembly sandbox so you can drill DataFrame and SQL logic anywhere with zero setup. When you want to inspect real JVM Catalyst physical plans, AQE skew splitting, shuffle spills, and Spark UI (:4040 / :18080), spin up this 3-container Docker Compose cluster on your laptop in 60 seconds.

1-Command Quickstart (after downloading files below)
docker compose up -d && docker exec dataarena-spark-master \
  /opt/spark/bin/spark-submit --master spark://spark-master:7077 \
  /opt/spark/work-dir/jobs/skew_aqe_demo.py

Works on Windows (WSL2 / Docker Desktop), macOS, and Linux.

Spark UI (:4040) Live Inspection PreviewAQE Skew Split Active
Stage / OperatorMedian · Max Task · Shuffle
BroadcastHashJoin / AQE1.2s · 1.8s · 18.4 MiB
Skewed Partition #0 (Split x8)0 Spill (Disk) · Coalesced
Web UIs: localhost:8080 · :4040 · :180801.5M rows
CapabilityBrowser Pyodide Sandbox (Here)Local Docker JVM Spark (Below)
PySpark DataFrame & SQL APIInstant in-browser testing across 243 problemsFull Apache Spark 3.5.3 JVM + Catalyst runtime
Spark UI (:4040) & History (:18080)Simulated architecture & lineage onlyLive Stage DAGs, Task Skew percentiles, GC & Spill bytes
AQE, Broadcast & Iceberg / Delta JARsLightweight Python shim (no JVM JARs)Real `--packages` Maven resolution (`delta-spark`, `iceberg-spark-runtime`)
File 1: docker-compose.yml

Spark Master + Worker + History Server with shared event logs

services:
  spark-master:
    image: apache/spark:3.5.3-python3
    container_name: dataarena-spark-master
    command: /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master
    environment:
      SPARK_MASTER_HOST: spark-master
      SPARK_MASTER_PORT: 7077
      SPARK_MASTER_WEBUI_PORT: 8080
    ports:
      - "8080:8080"   # Spark Master Web UI
      - "7077:7077"   # Spark Master RPC
      - "4040:4040"   # Live Application Spark UI (Jobs / Stages / SQL DAG)
    volumes:
      - ./jobs:/opt/spark/work-dir/jobs
      - ./warehouse:/opt/spark/work-dir/warehouse
      - ./spark-events:/tmp/spark-events

  spark-worker:
    image: apache/spark:3.5.3-python3
    container_name: dataarena-spark-worker
    depends_on:
      - spark-master
    command: /opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
    environment:
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 2g
    volumes:
      - ./jobs:/opt/spark/work-dir/jobs
      - ./warehouse:/opt/spark/work-dir/warehouse
      - ./spark-events:/tmp/spark-events

  spark-history:
    image: apache/spark:3.5.3-python3
    container_name: dataarena-spark-history
    command: /opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer
    environment:
      SPARK_HISTORY_OPTS: "-Dspark.history.fs.logDirectory=/tmp/spark-events"
    ports:
      - "18080:18080" # Spark History Server UI
    volumes:
      - ./spark-events:/tmp/spark-events
File 2: jobs/skew_aqe_demo.py

1.5M skewed-join workload to observe AQE skew splitting in Spark UI

# Save as ./jobs/skew_aqe_demo.py
from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
    .appName("DataArena-Skew-AQE-Lab")
    .config("spark.sql.adaptive.enabled", "true")
    .config("spark.sql.adaptive.skewJoin.enabled", "true")
    .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
    .config("spark.eventLog.enabled", "true")
    .config("spark.eventLog.dir", "/tmp/spark-events")
    .getOrCreate()
)

# 1. Generate skewed 1.5M-row fact table (80% traffic hits merchant_id = 1)
orders = (
    spark.range(0, 1_500_000)
    .withColumn(
        "merchant_id",
        F.when(F.col("id") % 10 < 8, F.lit(1)).otherwise((F.col("id") % 500) + 2)
    )
    .withColumn("amount", (F.col("id") % 250) + 1.0)
)

# 2. Dimension lookup table
merchants = (
    spark.range(1, 502)
    .withColumnRenamed("id", "merchant_id")
    .withColumn("tier", F.when(F.col("merchant_id") == 1, F.lit("MEGA")).otherwise(F.lit("SMB")))
)

# Force SortMergeJoin (bypass broadcast threshold) to watch AQE split skewed partition!
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")

joined = (
    orders.join(merchants, on="merchant_id", how="inner")
    .groupBy("tier")
    .agg(F.count("*").alias("tx_count"), F.round(F.sum("amount"), 2).alias("gmv"))
)

print("=== Catalyst Physical Plan ===")
joined.explain(mode="formatted")
joined.show()

spark.stop()
Step-by-Step Docker & spark-submit Commands
1. Create project folders & start the 3-container Spark cluster
mkdir -p spark-local/jobs spark-local/warehouse spark-local/spark-events && cd spark-local
# Place docker-compose.yml in spark-local/ and skew_aqe_demo.py in spark-local/jobs/
docker compose up -d
2. Submit PySpark job against real JVM Spark Master with Delta Lake package
docker exec -it dataarena-spark-master /opt/spark/bin/spark-submit \
  --master spark://spark-master:7077 \
  --packages io.delta:delta-spark_2.12:3.2.0 \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
  /opt/spark/work-dir/jobs/skew_aqe_demo.py
3. Submit with Apache Iceberg REST/Hadoop catalog locally
docker exec -it dataarena-spark-master /opt/spark/bin/spark-submit \
  --master spark://spark-master:7077 \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1 \
  --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.local.type=hadoop \
  --conf spark.sql.catalog.local.warehouse=/opt/spark/work-dir/warehouse \
  /opt/spark/work-dir/jobs/skew_aqe_demo.py
4. Inspect logs or tear down cleanly when finished
docker compose logs -f spark-worker
docker compose down -v

What to Inspect in Your Local Spark UIs

http://localhost:8080
Spark Master Web UI
Confirm 1 alive Worker (`2 cores`, `2 GiB RAM`) and watch Active vs Completed Applications appear in real time.
http://localhost:4040
Live Application UI (while job runs)
Open SQL / DataFrame tab -> expand the query DAG -> verify `AQEShuffleRead` splits skewed partition `merchant_id=1` instead of stalling 1 reducer.
http://localhost:18080
Spark History Server (after job stops)
Open Stages tab -> compare Min / Median / 75th / Max Task Duration + `Shuffle Write` + `Spill (Memory)` vs `Spill (Disk)` + GC Time (< 10%).
Resume Project Tip for Interviews

Save this setup in a public GitHub repo alongside one custom Medallion pipeline (CSV/JSON -> Bronze -> Deduped Silver Delta/Iceberg -> Gold Mart) plus screenshots of your Spark History Server DAG before and after AQE tuning. Next, pair it with our Performance Optimization Bible and 6-Round Interview Arena.