Week 15 • Day 5Week 15

Day 5:Agentic Token Budgets, Cost Caps & Oscillation Circuit Breakers

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 5: Agentic Token Budgets, Cost Caps & Oscillation Circuit Breakers

1. First-Principles Production Risk: The $500 Runaway Agent Loop

Unlike a single-turn RAG call whose cost is bounded by 1 LLM invocation, an autonomous multi-agent loop with tool retries can get stuck oscillating between two states (Agent A calls Agent B, Agent B calls Agent A) or repeatedly calling a failing tool, burning $50–$500 on a single user request!

Three Mandatory Production Circuit Breakers

  1. Per-Request Dollar & Token Budget (max_cost_usd, max_total_tokens): Costcumulative(T)=∑t=1T(It106⋅cin+Ot106⋅cout)≤Bmax⁡\text{Cost}_{\text{cumulative}}(T) = \sum_{t=1}^T \left( \frac{I_t}{10^6} \cdot c_{\text{in}} + \frac{O_t}{10^6} \cdot c_{\text{out}} \right) \le B_{\max}
  2. Hop-Count Cap (max_hops): Hard ceiling on total agent transitions per user request.
  3. Action-Signature Oscillation Detector: Hashes (agent_name, tool_name, canonical_args). If the exact same signature appears ≥Rmax⁡\ge R_{\max} times (e.g., 2 times) within a sliding window, trip the circuit breaker immediately.

In Simple English: Putting a Credit Limit on Your Agents (Cost Caps & Circuit Breakers)

Never deploy an autonomous loop without hard spending brakes! Look at Costcumulative(T)\text{Cost}_{\text{cumulative}}(T):

  • Every turn tt adds both Prompt Input Tokens ItI_t (which grow larger on every step as chat history piles up!) AND Output Tokens OtO_t.
  • If an agent gets stuck retrying broken searches for 20 steps, cumulative cost grows quadratically!
  • Production Safety Brakes:
    1. max_steps cap (e.g., 10 turns hard stop).
    2. max_budget_usd accumulator checked before every LLM call.
    3. Oscillation Detector: If the agent flips Call A -> Call B -> Call A -> Call B, halt immediately and summarize partial findings!