Back to PlaygroundWatermark: 14s · Max Event Time: 24s
Interactive Spark Structured Streaming & Kafka Watermark Lab

Event-Time, Watermark & Late-Data Visualizer

Tune tumbling vs sliding windows and watermark lag below. Watch when late mobile events update open window state vs get evicted to Dead-Letter Queues once event_time < max_event_time - watermark.

Watermark & Window Parameters
On-Time Events
5
Late Accepted
1
Dropped (DLQ)
1
Open State Buckets
2
Active & Finalized Window State Store
[0s, 10s)FINALIZED & EVICTED
Count: 3 · Sum: $155 (1 late)
[10s, 20s)OPEN IN STATE STORE
Count: 1 · Sum: $120
[20s, 30s)OPEN IN STATE STORE
Count: 2 · Sum: $205
Arrival Order & Watermark Evaluation Log
#1 e1 (event_time=2s, $45)On-time arrival (2s >= watermark 0s)
#2 e2 (event_time=7s, $80)On-time arrival (7s >= watermark 0s)
#3 e3 (event_time=13s, $120)On-time arrival (13s >= watermark 0s)
#4 e4 (event_time=6s, $30)Out-of-order (6s < max 13s) yet >= watermark 3s — state updated
#5 e5 (event_time=22s, $95)On-time arrival (22s >= watermark 3s)
#6 e6 (event_time=8s, $60)Event time 8s < active watermark 12s (evicted window state -> DLQ)
#7 e7 (event_time=24s, $110)On-time arrival (24s >= watermark 12s)
Equivalent PySpark Structured Streaming + Kafka Code
from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("WatermarkedCheckoutStream").getOrCreate()

events = (
    spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "checkout_events")
    .load()
)

aggregated = (
    events
    .withWatermark("event_time", "10 seconds")
    .groupBy(
        F.window("event_time", "10 seconds", "10 seconds"),
        F.col("key")
    )
    .agg(
        F.count("*").alias("event_count"),
        F.sum("amount").alias("total_amount_usd")
    )
)

query = (
    aggregated.writeStream
    .outputMode("update")
    .format("console")
    .start()
)