Instagram Live: The Thundering Herd & Redis Cache Stampede
Meta Forensic IncidentStaff Production Engineer (L6)$340,000 - $430,000 / yr
Instagram Live: The Thundering Herd & Redis Cache Stampede
“Welcome to Meta Production Engineering. You are in our War Room. An Instagram Live broadcast went viral, causing the cached live-stream metadata key to expire. Our primary MySQL database is locked with 10,000 waiting threads. Propose a distributed caching architecture with probabilistic early expiration, mutex locking, and connection pool protection.”
During a live broadcast with 12 million concurrent viewers, the Redis video metadata cache expired simultaneously. 850,000 requests per second slammed the primary MySQL database cluster, causing replica lag of 480 seconds and database connection exhaustion.