Week 18 • Day 3Week 18
Day 3:Semantic Prompt Caching (Cosine Similarity >= 0.95)
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 3: Semantic Prompt Caching (Cosine Similarity >= 0.95)
1. First-Principles Problem: Why Exact String Hashing Fails for AI
In conventional Web caching (Redis / Memcached), cache keys are computed via cryptographic string hashing: If a second user asks: The SHA-256 hashes are completely disjoint! The system incurs a cache miss, burning 1,500ms of latency and paying the LLM provider for duplicate token generation.
In enterprise IT and incident response, 30% to 55% of all daily questions are semantic paraphrases of recurring questions.
2. Mathematical Formulation of Semantic Caching
Instead of string hashing, we store entries in a Vector Semantic Cache:
Architecture Flow Diagram
Incoming Query q->Emb[Compute Normalized Embedding e_q
Search[Search Cached Vectors (Cosine Sim)
Check{Max Sim >= 0.95?}
|Yes (Cache Hit)| Hit[Return Cached Answer (< 10ms, $0.00 Cost)
|No (Cache Miss)| Miss[Forward to LLM (1500ms, Full Cost)
Store[Store (e_q, answer, TTL) in Cache
Cosine Similarity on Unit-Normalized Vectors:
Threshold Selection Dynamics:
- (Ultra Conservative): High precision, low hit rate (~15%). Misses obvious synonyms.
- (The Industry Sweet Spot): Flags genuine paraphrases with factual semantic alignment while preventing drift.
- (Dangerous Semantic Drift): False positives begin occurring (e.g., confusing "How do I scale up?" with "How do I scale down?").
Economic & Latency Impact:
In Simple English: Caching by Meaning, Not Exact Spelling (Semantic Cache)
Suppose User A asks: "Why is checkout crashing?" and User B asks 1 minute later: "Why did checkout-api crash?"
- Exact Hash Cache (
SHA-256): Misses! One different character breaks exact matching, so you pay LLM cost twice. - Semantic Prompt Cache: Embeds both questions into vectors and checks . When meaning matches (
0.97 >= 0.95), return the saved answer in6 msat$0.0000! Remember from Week 12: ALWAYS scope cache lookups pertenant_id + roleso Company B never reads Company A's cached secrets!