Google Cloud Dataflow: Apache Beam Unified Batch & Streaming

[!NOTE] Google Cloud Dataflow is a fully managed, serverless execution service for Apache Beam pipelines. Beam provides a unified programming model where the exact same pipeline code executes over bounded (batch) or unbounded (streaming) datasets.

1. Apache Beam Core Primitives

• Pipeline: Manages the directed acyclic graph (DAG) of transformations.
• PCollection: Immutable distributed dataset (bounded or unbounded).
• PTransform: Data transformation step applied to a PCollection.
• ParDo & DoFn: Core element-wise processing function (analogous to Spark's map / flatMap).

Staff Architecture: Cloud Dataflow (Apache Beam), Runner v2, Streaming Engine & Watermarks

• Dataflow Shuffle Service & Streaming Engine: Never run large batch shuffles (GroupByKey / CoGroupByKey) or Stateful streaming windows on worker local disks—enable Dataflow Shuffle (batch) and Streaming Engine (streaming) to offload shuffle & state storage to Google’s managed service, shrinking worker RAM needs and enabling fast horizontal Worker Autoscaling.
• Dataflow Runner v2 (Portable Beam): Use Runner v2 whenever mixing Python transforms with Java/Cross-Language I/O connectors (e.g., KafkaIO, BigQuery Storage Write API), using custom Docker worker containers for clean dependency reproducibility.
• Watermarks, Triggers & Late Data: Streaming pipelines must explicitly configure withAllowedLateness(...) and Dead-Letter side outputs (TupleTag)—otherwise malformed Pub/Sub JSON messages trap an unbounded worker retry loop forever, freezing the watermark and halting downstream windowed outputs.