Loading Academy Hub...
Design enterprise-grade AWS Lakehouse storage: S3 request rate limits (3,500 PUT / 5,500 GET per prefix), prefix hash sharding, partition layout design (year/month/day/hour), lifecycle transition policies, and S3 Object Lock governance.
[!NOTE] Amazon S3 is the foundational storage layer of modern AWS Lakehouses. While virtually unlimited in capacity, S3 enforces per-prefix request rate limits: 3,500 PUT/POST/DELETE requests per second and 5,500 GET/HEAD requests per second per prefix.
In S3, a prefix is anything between the bucket name and the object name.
For example:
s3://lakehouse-prod-bucket/landing/finance/year=2026/month=03/txns.parquet
Here, the prefix is landing/finance/year=2026/month=03/.
s3://lakehouse/landing/hash=a1b2/year=2026/month=03/
s3://lakehouse/landing/hash=c3d4/year=2026/month=03/s3://enterprise-lakehouse-prod/
├── bronze/ (Raw Append-Only)
│ └── sales/year=2026/month=03/day=01/file1.parquet
├── silver/ (Cleaned Delta / Iceberg)
│ └── sales/department=finance/year=2026/file2.parquet
└── gold/ (Aggregated Reporting & Bedrock AI Vectors)
└── daily_summary/year=2026/file3.parquet
3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix. High-concurrency Spark jobs writing unpartitioned UUID directories or Hive-style dt=YYYY-MM-DD hot prefixes can still trip 503 SlowDown if a single micro-partition fanout exceeds KMS key quotas or a tight prefix rate.kms:Decrypt or kms:GenerateDataKey API call against AWS KMS (default soft limit 5,500–30,000 req/s per region), inflating costs and causing intermittent Spark throttling. Enabling S3 Bucket Keys cuts KMS API traffic by up to 99%.fs.s3a.committer.name=magic with S3 strong read-after-write consistency) instead of FileOutputCommitter v1/v2 so Spark avoids expensive COPY+DELETE object renames during task commit.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import boto3
from boto3.s3.transfer import TransferConfig
# Configure optimal high-throughput transfer parameters
s3_client = boto3.client('s3', region_name='us-east-1')
config = TransferConfig(
multipart_threshold=100 * 1024 * 1024, # 100MB
max_concurrency=20, # 20 parallel threads
multipart_chunksize=50 * 1024 * 1024, # 50MB per chunk
use_threads=True
)
# Upload 20GB dataset utilizing multiple S3 prefix connections
s3_client.upload_file(
Filename='/tmp/large_lakehouse_batch.parquet',
Bucket='enterprise-lakehouse-prod',
Key='bronze/finance/year=2026/month=03/batch_01.parquet',
Config=config
)
print("Multipart upload successfully completed with 20 parallel threads.")botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the GetObject operation: Please reduce your request rate.