Loading Academy Hub...
Master AWS failure forensics and operational runbooks: Amazon CloudWatch metrics, CloudTrail audit logs, Athena QueryExceededMemoryLimit triage, Glue DPU OOM crashes, and S3 503 SlowDown resolution.
[!NOTE] Cloud data pipelines operate in multi-tenant environments with dynamic quotas. This module provides verified incident runbooks for the most common AWS data platform production outages.
QueryExceededMemoryLimit: Occurs when aggregations or joins require more RAM than the allocated Athena cluster worker tier.SlowDown (HTTP 503): Request rate limit breached on a single S3 prefix.Container Killed by YARN for exceeding memory limits: Executor Out-Of-Memory caused by large skew partitions or untruncated datasets.503 SlowDown, Glue OutOfMemory & Cross-Account KMS ErrorsExitCode 137 / Container Killed: In AWS Glue, G.1X provides 16 GB RAM (4 vCPU) and G.2X provides 32 GB RAM (8 vCPU). If Driver OOM occurs during collect(), toPandas(), or broadcast joins, either bump to G.2X, increase --conf spark.driver.memoryOverhead, or remove driver-side collect() calls.AccessDenied Even With Bucket Policy: Almost every cross-account AccessDenied on KMS-encrypted S3 data happens because both the IAM Role policy and the KMS Key Policy must explicitly grant kms:Decrypt / kms:GenerateDataKey—or because S3 Block Public Access / Bucket Owner Enforced ACLs conflict with legacy bucket-owner-full-control writers.HIVE_CANNOT_OPEN_SPLIT / Query exhausted resources: Usually triggered by unsplit-able huge .json.gz / .csv.gz files (>1GB gzip cannot be parallelized across Athena workers) or massive ORDER BY without LIMIT forcing single-node final sort.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import boto3
from datetime import datetime, timedelta
logs = boto3.client('logs', region_name='us-east-1')
query = """
fields @timestamp, @message
| filter @message like /Container killed/ or @message like /OutOfMemory/ or @message like /SlowDown/
| sort @timestamp desc
| limit 50
"""
response = logs.start_query(
logGroupName='/aws-glue/jobs/error',
startTime=int((datetime.now() - timedelta(hours=6)).timestamp()),
endTime=int(datetime.now().timestamp()),
queryString=query
)
print(f"Started CloudWatch query {response['queryId']} to triage production crashes.")Query failed: Query exceeded memory limit for a query on this cluster. Try optimizing your query or reducing concurrent queries.