Loading Academy Hub...
Master enterprise failure forensics across GCP data platforms: Google Cloud Operations Suite (Cloud Logging/Monitoring), BigQuery `resourcesExceeded` triage, and Dataflow pipeline drain.
[!NOTE] When production pipelines fail, data engineers rely on Google Cloud Operations Suite (formerly Stackdriver) to isolate root causes and restore operations.
resourcesExceeded: Occurs when a query demands more shuffle RAM or slot resources than the project limit.OOM Killed): Memory leak or skewed GroupByKey in Apache Beam pipeline.429 RateLimitExceeded: Too many mutations (creates/deletes) under a single GCS bucket prefix.Resources exceeded, Concurrent update Contention & Dataflow Hot KeysResources exceeded during query execution (ORDER BY / COUNT(DISTINCT) / Correlated Subqueries): Most often triggered by an un-limited ORDER BY, non-equi JOIN (BETWEEN timestamp ranges causing Cartesian blowups), or giant single-partition ARRAY_AGG / WINDOW over unbounded rows. Fix via explicit LIMIT, range-bucketed joins, APPROX_COUNT_DISTINCT, or pre-aggregating before windowing.Could not serialize access to table due to concurrent update: BigQuery enforces a hard quota of concurrent DML (UPDATE/DELETE/MERGE) mutations hitting the same table (20 queued mutations max). Batch streaming/micro-ETL writes into Storage Write API appends + periodic 15-minute consolidated MERGE runs instead of firing concurrent per-file MERGE statements.Hot Key in GroupByKey): Use Combine.perKey with .withHotKeyFanout(...) so partial aggregations happen across multiple random sub-shards before the final merge on one worker.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
from google.cloud import logging
client = logging.Client()
filter_str = """
resource.type="bigquery_resource"
severity>=ERROR
timestamp >= "2026-03-01T00:00:00Z"
"""
for entry in client.list_entries(filter_=filter_str, page_size=20):
print(f"[{entry.timestamp}] {entry.payload.get('message', entry.payload)}")google.api_core.exceptions.Forbidden: 403 Resources exceeded during query execution: The query could not be executed in the allotted memory. Peak shuffle memory exceeded limit.