Week 17 • Day 4Week 17
Day 4:API Security, Token-Bucket Rate Limiting & Spend Quotas
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 4: API Security, Token-Bucket Rate Limiting & Spend Quotas
1. First-Principles Risk: The "Denial-of-Wallet" (DoW) Attack
In conventional software engineering, a Denial-of-Service (DoS) attack exhausts CPU or network bandwidth. In AI Engineering, an unmetered API endpoint exposes the company to a Denial-of-Wallet (DoW) attack:
- Suppose a bot scrapes an unauthenticated
/v1/copilot/queryendpoint with10requests/second for4hours ($144,000$ requests). - If each request processes
2,000tokens using a frontier model at \5.00\text{ / 1M input tokens}: $$\text{Cost} = 144,000 \times \frac{2,000}{1,000,000} \times \5.00 = $1,440.00 \text{ in 4 hours!}$$
2. Mathematical Formulation of the Token-Bucket Algorithm
The Token-Bucket algorithm enables smooth burst handling while strictly enforcing a sustainable average rate:
Architecture Flow Diagram
Continuous Refill (+r tokens/sec)->Bucket[Token Bucket (Capacity C)
Incoming Request (Cost = 1)->Gate{Tokens >= Cost?}
-.
|Yes| Allow[Allow Request (Deduct Cost from Bucket)
|No| Deny[HTTP 429 Too Many Requests (Retry-After)
State Variables:
- : Bucket Capacity (maximum burst allowance)
- : Refill Rate (tokens added per second)
- : Timestamp of the last processed request
- : Available tokens at
Refill Equation at Request Time :
Decision Rule:
3. Dual-Layer Defense: RPM + Daily Tenant Spend Cap
Production AI gateways enforce two separate barriers:
- Layer 1: Transient Velocity Gate (Token Bucket): Enforces Requests Per Minute (e.g., 60 RPM per tenant).
- Layer 2: Cumulative Dollar Spend Cap: Enforces total daily API spend per tenant (e.g., $25.00/day). If the tenant's cumulative spend reaches the limit, all subsequent requests are rejected with
HTTP 402 Payment RequiredorHTTP 429 Quota Exceededuntil the next UTC billing window.
In Simple English: How a Token Bucket Stops Runaway API Bills (HTTP 429)
Think of the Token Bucket like an arcade coin jar holding up to coins, with a refill pipe dripping new coins per second:
- When a user sends a chat request, take 1 coin out of the jar.
- Short bursts (5 clicks quickly) succeed smoothly because coins saved up in the jar absorb the spike!
- If an automated scraper spams 100 requests in 1 second, the jar hits 0 coins instantly return
HTTP 429 Too Many Requestswith headerRetry-AfterBEFORE calling your expensive LLM!