Back to PlaygroundWatermark: 14s · Max Event Time: 24s
Interactive Spark Structured Streaming & Kafka Watermark Lab
Event-Time, Watermark & Late-Data Visualizer
Tune tumbling vs sliding windows and watermark lag below. Watch when late mobile events update open window state vs get evicted to Dead-Letter Queues once event_time < max_event_time - watermark.
Watermark & Window Parameters
On-Time Events
5
Late Accepted
1
Dropped (DLQ)
1
Open State Buckets
2
Active & Finalized Window State Store
[0s, 10s)FINALIZED & EVICTED
Count: 3 · Sum: $155 (1 late)
[10s, 20s)OPEN IN STATE STORE
Count: 1 · Sum: $120
[20s, 30s)OPEN IN STATE STORE
Count: 2 · Sum: $205
Arrival Order & Watermark Evaluation Log
#1 e1 (event_time=2s, $45)On-time arrival (2s >= watermark 0s)
#2 e2 (event_time=7s, $80)On-time arrival (7s >= watermark 0s)
#3 e3 (event_time=13s, $120)On-time arrival (13s >= watermark 0s)
#4 e4 (event_time=6s, $30)Out-of-order (6s < max 13s) yet >= watermark 3s — state updated
#5 e5 (event_time=22s, $95)On-time arrival (22s >= watermark 3s)
#6 e6 (event_time=8s, $60)Event time 8s < active watermark 12s (evicted window state -> DLQ)
#7 e7 (event_time=24s, $110)On-time arrival (24s >= watermark 12s)
Equivalent PySpark Structured Streaming + Kafka Code
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
spark = SparkSession.builder.appName("WatermarkedCheckoutStream").getOrCreate()
events = (
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "checkout_events")
.load()
)
aggregated = (
events
.withWatermark("event_time", "10 seconds")
.groupBy(
F.window("event_time", "10 seconds", "10 seconds"),
F.col("key")
)
.agg(
F.count("*").alias("event_count"),
F.sum("amount").alias("total_amount_usd")
)
)
query = (
aggregated.writeStream
.outputMode("update")
.format("console")
.start()
)