Week 20 • Day 3Week 20

Day 3:Semantic Cache & OpenTelemetry Tracing Integration

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 3: Semantic Cache & OpenTelemetry Tracing Integration

1. First-Principles: Zero-Cost Fast-Path Execution

When an incoming query reaches the microservice:

  1. The Semantic Cache performs a vector similarity lookup against cached embeddings.
  2. If similarity exceeds θ≥0.94\theta \ge 0.94, the service bypasses the LLM entirely:
    • Emits cached tokens immediately
    • Incurs \0.00$ token cost
    • Achieves <10ms< 10\text{ms} turnaround time
    • Records an OTel span and updates Prometheus cache hit metrics
python
cache_res = self.cache.lookup(payload.query) if cache_res.hit: # Zero-cost return fast-path return cached_stream

In Simple English: Semantic Cache + Trace Recording Fast Path (Month 5 Capstone Step 3)

Look at what happens in the 10-millisecond fast path:

  1. Incoming user prompt checks the Tenant-Scoped Semantic Cache (cosine >= 0.95).
  2. On Cache Hit, emit cached SSE tokens in <10 ms, incur $0.00 LLM spend, increment cache_hits_total, and tag the OpenTelemetry trace with cache.hit = true!
  3. On Cache Miss, route to primary/fallback LLM, stream live tokens, log cost, and store the clean reply in cache for future similar questions!

Interactive Sandbox: Run semantic_cache_fastpath.py and its unit tests in Tab 2 (Codes) & Tab 4 (Test & Troubleshooting) directly inside your browser!