Week 17 • Day 4Week 17

Day 4:API Security, Token-Bucket Rate Limiting & Spend Quotas

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 4: API Security, Token-Bucket Rate Limiting & Spend Quotas

1. First-Principles Risk: The "Denial-of-Wallet" (DoW) Attack

In conventional software engineering, a Denial-of-Service (DoS) attack exhausts CPU or network bandwidth. In AI Engineering, an unmetered API endpoint exposes the company to a Denial-of-Wallet (DoW) attack:

  • Suppose a bot scrapes an unauthenticated /v1/copilot/query endpoint with 10 requests/second for 4 hours ($144,000$ requests).
  • If each request processes 2,000 tokens using a frontier model at \5.00\text{ / 1M input tokens}: $$\text{Cost} = 144,000 \times \frac{2,000}{1,000,000} \times \5.00 = $1,440.00 \text{ in 4 hours!}$$

2. Mathematical Formulation of the Token-Bucket Algorithm

The Token-Bucket algorithm enables smooth burst handling while strictly enforcing a sustainable average rate:

Architecture Flow Diagram
Continuous Refill (+r tokens/sec)->Bucket[Token Bucket (Capacity C)
Incoming Request (Cost = 1)->Gate{Tokens >= Cost?}
-.
|Yes| Allow[Allow Request (Deduct Cost from Bucket)
|No| Deny[HTTP 429 Too Many Requests (Retry-After)

State Variables:

  • CC: Bucket Capacity (maximum burst allowance)
  • rr: Refill Rate (tokens added per second)
  • tlastt_{\text{last}}: Timestamp of the last processed request
  • BcurrentB_{\text{current}}: Available tokens at tlastt_{\text{last}}

Refill Equation at Request Time tt:

B(t)=min⁡(C,Bcurrent+r⋅(t−tlast))B(t) = \min\Big(C, B_{\text{current}} + r \cdot (t - t_{\text{last}})\Big)

Decision Rule:

If B(t)≥1  ⟹  Bnew=B(t)−1,ALLOW\text{If } B(t) \ge 1 \implies B_{\text{new}} = B(t) - 1, \quad \text{ALLOW} If B(t)<1  ⟹  DENY (HTTP 429),Retry-After=1−B(t)r seconds\text{If } B(t) < 1 \implies \text{DENY (HTTP 429)}, \quad \text{Retry-After} = \frac{1 - B(t)}{r} \text{ seconds}


3. Dual-Layer Defense: RPM + Daily Tenant Spend Cap

Production AI gateways enforce two separate barriers:

  1. Layer 1: Transient Velocity Gate (Token Bucket): Enforces Requests Per Minute (e.g., 60 RPM per tenant).
  2. Layer 2: Cumulative Dollar Spend Cap: Enforces total daily API spend per tenant (e.g., $25.00/day). If the tenant's cumulative spend reaches the limit, all subsequent requests are rejected with HTTP 402 Payment Required or HTTP 429 Quota Exceeded until the next UTC billing window.

In Simple English: How a Token Bucket Stops Runaway API Bills (HTTP 429)

Think of the Token Bucket like an arcade coin jar holding up to C=10C = 10 coins, with a refill pipe dripping r=2r = 2 new coins per second:

  • When a user sends a chat request, take 1 coin out of the jar.
  • Short bursts (5 clicks quickly) succeed smoothly because coins saved up in the jar absorb the spike!
  • If an automated scraper spams 100 requests in 1 second, the jar hits 0 coins →\rightarrow instantly return HTTP 429 Too Many Requests with header Retry-After BEFORE calling your expensive LLM!