Google Cloud Pub/Sub, Eventarc & BigQuery Storage Write API

[!NOTE] Google Cloud Pub/Sub provides horizontally scalable, globally distributed asynchronous messaging. Combined with the BigQuery Storage Write API, data engineers achieve sub-second ingestion directly into BigQuery tables with exactly-once delivery guarantees.

1. Storage Write API vs Legacy streamingBuffer

• Legacy tabledata.insertAll: Row-by-row HTTP JSON inserts; expensive and subject to strict rate limits.
• BigQuery Storage Write API: Streams binary Protocol Buffers (Protobuf) over gRPC.
  • Supports Default Stream (at-least-once, zero cost) and Committed Stream (exactly-once with offset tracking).
  • Up to 70% cheaper than legacy streaming.

Staff Architecture: Cloud Pub/Sub -> BigQuery Subscriptions vs Storage Write API

• Zero-Code Pub/Sub BigQuery Subscriptions: For simple append-only JSON/Avro/Protobuf event streams with schema validation, Pub/Sub BigQuery Subscriptions write directly from Pub/Sub into BigQuery with zero Dataflow workers—slashing streaming infrastructure cost by 60–80%.
• Pub/Sub Exactly-Once Delivery vs End-to-End Idempotency: Pub/Sub guarantees at-least-once delivery globally (with optional enable_exactly_once_delivery inside single-region pull leases). Still design BigQuery downstream MERGE / Storage Write API deduplication (Change Data Capture or ROW_NUMBER() OVER (PARTITION BY event_id)), because upstream mobile/API publishers can always emit duplicate retries.
• Legacy insertAll Deprecation Path: Migrate legacy tabledata.insertAll streaming code to the BigQuery Storage Write API (Committed or Pending streams)—higher throughput, free default committed-stream pricing (2 TB/month free), strict protobuf schema enforcement, and optional exactly-once offsets.