Week 17 • Day 6Week 17
Day 6:Concurrency Profiling, Little's Law (L = lambda W) & Latency Percentiles
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 6: Concurrency Profiling, Little's Law (L = lambda W) & Latency Percentiles
1. First-Principles Queuing Theory: Little's Law
Every AI backend engineer must master Little's Law from operational queuing theory: where:
- : Average number of concurrent in-flight requests inside the system.
- : Arrival rate of requests (throughput, in requests / second).
- : Average turnaround time / latency per request (in seconds).
Production AI Sizing Example:
Suppose an enterprise team expects during an incident, and average agentic turnaround time is .
- In a synchronous WSGI server with 16 threads, the queue fills up instantly, and 84 requests wait in an OS backlog queue.
- In an asynchronous ASGI server, 100 async tasks are suspended on I/O waiting for LLM tokens, consuming just a few megabytes of RAM.
2. Why Arithmetic Mean (Average) Is Dangerous in AI
In traditional web applications, response times follow a roughly bell-shaped distribution. In AI systems, response distributions are heavily bimodal or long-tailed:
- Simple queries (cached or single-turn): $400\text{ms}$
- Complex multi-step agent tool calls: $6,500\text{ms}$
If 90% of queries take $400\text{ms}6,500\text{ms}$: Reporting an "average latency of ~1 second" masks the fact that 1 out of every 10 users is waiting 6.5 seconds!
Industry Standard Percentile Metrics:
- p50 (Median): 50% of requests are faster than this threshold.
- p95: 95% of requests are faster than this threshold (standard SLA commit).
- p99: Tail latency experienced by 1% of users (often the highest-paying enterprise users running complex multi-file workflows).
In Simple English: Little's Law & Why Averages Hide Disaster (P95 / P99)
- Little's Law ( — Coffee Shop Seating Rule):
- If customers walk in every second, and each stays seconds, how many chairs () must your café have open simultaneously?
- $40 \times 2.5 = \mathbf{100\text{ active open connections}}L = \lambda W$, never guesses.
- Why Average Latency Lies:
- Suppose 9 users wait
400msand 1 VIP user waits6,500ms. Average =1,010ms("looks fine!"), yet your slowest 10% of users (P90 / P99) wait 6.5 seconds and rage-quit! Senior SRE dashboards always alert onP95&P99, never averages.
- Suppose 9 users wait