Streaming: Watermark & Late-Event Deduplication

Hard
Streaming
Watermark
Deduplication
Late Data
Uber
Stripe
Databricks

Streaming: Watermark & Late-Event Deduplication

Given dfs['events'] (event_id, event_ts, amount), protect downstream state from late retries and unbounded memory growth:

  1. Call .withWatermark('event_ts', '15 minutes').
  2. Drop late events arriving before the active watermark cutoff (event_ts >= '2026-05-01 11:45:00').
  3. Deduplicate identical events with .dropDuplicates(['event_id']).
  4. Return event_id, event_ts, amount ordered by event_id.
Browser Engine: Evaluates PySpark DataFrame & SQL logic locally in Pyodide WebAssembly.

Input data

0 DataFrames

These DataFrames are passed to solve(spark, dfs). Access one with dfs['name'].

Browser engine
Loading Python editor…
Click Run to try the sample cases, or Submit to check the hidden cases. First run caches the browser Python engine (~8–15s once); all code stays on your device.