Google Cloud Dataproc: Ephemeral Clusters, Spot VMs & BigQuery Connector

[!NOTE] Google Cloud Dataproc provides fast (sub-90-second provisioning) managed Apache Spark and Hadoop clusters. The production pattern uses Ephemeral Clusters: create a cluster, run a single job, and immediately delete the cluster to eliminate idle costs.

1. Ephemeral Clusters & Preemptible / Spot VMs

• Compute Isolation: Running one cluster per job prevents memory leaks and JVM garbage collection drift.
• Spot / Preemptible VMs: Secondary worker nodes run on Spot VMs at up to 80% discount.
  • If Google reclaims a Spot worker, Spark re-computes only the lost partition without failing the overall job.
• BigQuery Spark Connector: Reads BigQuery tables using the BigQuery Storage Read API (gRPC streams in Arrow format) for 10x faster reads than traditional REST APIs.

Staff Architecture: Dataproc Serverless vs Ephemeral Clusters, GCS Connector & BigLake

• Ephemeral vs Dataproc Serverless (Batches): Never leave idle Dataproc GCE clusters sitting overnight for cron PySpark jobs. Run Dataproc Serverless Batches (zero cluster provisioning, auto-scaling executors) or ephemeral Workflow Templates that spin up per DAG run, read/write GCS directly, and self-destruct immediately on completion.
• BigQuery Spark Connector (Storage Read/Write API): Avoid legacy GCS-staging export mode when reading/writing massive BigQuery tables from Spark—use the spark-bigquery-connector v2 (Storage Read API + DIRECT Storage Write API) for parallel Apache Arrow streams directly into Spark partitions.
• Preemptible/Spot Secondary Workers & EFM: Pair Standard Primary workers (2–4) with cheap Spot Secondary Workers and Enhanced Flexibility Mode (EFM) so shuffle data stays anchored on reliable primary nodes (spark.dataproc.enhanced.execution.enabled) when GCP reclaims Spot VMs.