Loading Academy Hub...
Design resilient serverless AWS ingestion pipelines: S3 prefix hash partitioning (5,500 req/sec limits), SQS buffering with Dead-Letter Queues (DLQ), AWS Glue Studio DynamicFrames with groupFiles memory mitigation, Athena Partition Projection, and Bedrock Titan Vector embeddings.
Inspect console configs, simulate Happy Path vs Fault-Injection & DLQ recovery, or copy production IaC/code.
High-throughput landing bucket with prefix hash sharding (5,500 req/sec limit).
[!NOTE] Building enterprise AWS data pipelines requires orchestrating decoupled serverless primitives: Amazon S3 for object storage, Amazon SQS for ingestion buffering, AWS Glue for serverless Apache Spark transformation, Amazon Athena for query serving, and Amazon Bedrock for multi-modal vector generation.
ObjectCreated events directly to AWS Lambda or Glue in high-throughput environments. Buffer via SQS queues with Dead-Letter Queues (DLQ) to prevent concurrency exhaustion.groupFiles: 'inPartition' and groupSize: 134217728 (128MB) when reading millions of tiny JSON objects to prevent YARN container OOM (error 137).MSCK REPAIR TABLE commands and AWS Glue Crawler latency.Embed Gold Lakehouse textual summaries using Amazon Bedrock Titan (amazon.titan-embed-text-v2:0) into Amazon OpenSearch Serverless for hybrid RAG search.
Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
glueContext = GlueContext(SparkContext.getOrCreate())
spark = glueContext.spark_session
# Ingest landed JSON files with automatic small-file grouping (128MB chunks)
dynamic_frame = glueContext.create_dynamic_frame.from_options(
connection_type="s3",
connection_options={
"paths": ["s3://enterprise-lakehouse-prod-landing/landing/year=2026/month=03/"],
"recurse": True,
"groupFiles": "inPartition",
"groupSize": "134217728" # 128 MB grouping
},
format="json"
)
# Convert to DataFrame, apply transformations, and write to Parquet Gold table
df = dynamic_frame.toDF()
df_cleaned = df.dropDuplicates(["transaction_id"]).filter(df["amount"] > 0)
df_cleaned.write \
.format("parquet") \
.mode("append") \
.partitionBy("year", "month") \
.save("s3://enterprise-lakehouse-prod-gold/analytics/transactions/")
print("Glue job successfully compacted and wrote partitioned Parquet records.")Container [pid=18412,containerID=container_1614_01_000002] is running 18239014 bytes beyond physical memory-limit. Killing container.