Week 20 • Day 3Week 20
Day 3:Semantic Cache & OpenTelemetry Tracing Integration
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 3: Semantic Cache & OpenTelemetry Tracing Integration
1. First-Principles: Zero-Cost Fast-Path Execution
When an incoming query reaches the microservice:
- The Semantic Cache performs a vector similarity lookup against cached embeddings.
- If similarity exceeds , the service bypasses the LLM entirely:
- Emits cached tokens immediately
- Incurs \0.00$ token cost
- Achieves turnaround time
- Records an OTel span and updates Prometheus cache hit metrics
python
cache_res = self.cache.lookup(payload.query)
if cache_res.hit:
# Zero-cost return fast-path
return cached_stream
In Simple English: Semantic Cache + Trace Recording Fast Path (Month 5 Capstone Step 3)
Look at what happens in the 10-millisecond fast path:
- Incoming user prompt checks the Tenant-Scoped Semantic Cache (
cosine >= 0.95). - On Cache Hit, emit cached SSE tokens in
<10 ms, incur$0.00LLM spend, incrementcache_hits_total, and tag the OpenTelemetry trace withcache.hit = true! - On Cache Miss, route to primary/fallback LLM, stream live tokens, log cost, and store the clean reply in cache for future similar questions!
Interactive Sandbox: Run
semantic_cache_fastpath.pyand its unit tests in Tab 2 (Codes) & Tab 4 (Test & Troubleshooting) directly inside your browser!