Week 18 • Day 3Week 18

Day 3:Semantic Prompt Caching (Cosine Similarity >= 0.95)

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 3: Semantic Prompt Caching (Cosine Similarity >= 0.95)

1. First-Principles Problem: Why Exact String Hashing Fails for AI

In conventional Web caching (Redis / Memcached), cache keys are computed via cryptographic string hashing: Key=SHA256("Why is the checkout pod crashing?")\text{Key} = \text{SHA256}(\text{"Why is the checkout pod crashing?"}) If a second user asks: Query 2="Why is checkout-api pod in CrashLoopBackOff?"\text{Query 2} = \text{"Why is checkout-api pod in CrashLoopBackOff?"} The SHA-256 hashes are completely disjoint! The system incurs a cache miss, burning 1,500ms of latency and paying the LLM provider for duplicate token generation.

In enterprise IT and incident response, 30% to 55% of all daily questions are semantic paraphrases of recurring questions.


2. Mathematical Formulation of Semantic Caching

Instead of string hashing, we store entries in a Vector Semantic Cache:

Architecture Flow Diagram
Incoming Query q->Emb[Compute Normalized Embedding e_q
Search[Search Cached Vectors (Cosine Sim)
Check{Max Sim >= 0.95?}
|Yes (Cache Hit)| Hit[Return Cached Answer (< 10ms, $0.00 Cost)
|No (Cache Miss)| Miss[Forward to LLM (1500ms, Full Cost)
Store[Store (e_q, answer, TTL) in Cache

Cosine Similarity on Unit-Normalized Vectors:

sim(q,ci)=eq⋅eci∥eq∥∥eci∥=∑j=1Deq,j⋅eci,j\text{sim}(q, c_i) = \frac{\mathbf{e}_q \cdot \mathbf{e}_{c_i}}{\|\mathbf{e}_q\| \|\mathbf{e}_{c_i}\|} = \sum_{j=1}^D e_{q, j} \cdot e_{c_i, j}

Threshold θ\theta Selection Dynamics:

  • θ≥0.98\theta \ge 0.98 (Ultra Conservative): High precision, low hit rate (~15%). Misses obvious synonyms.
  • θ=0.95\theta = 0.95 (The Industry Sweet Spot): Flags genuine paraphrases with >99.2%> 99.2\% factual semantic alignment while preventing drift.
  • θ≤0.88\theta \le 0.88 (Dangerous Semantic Drift): False positives begin occurring (e.g., confusing "How do I scale up?" with "How do I scale down?").

Economic & Latency Impact:

Hit Latency≈6msvs.Miss Latency≈1,800ms(300× faster!)\text{Hit Latency} \approx 6\text{ms} \quad \text{vs.} \quad \text{Miss Latency} \approx 1,800\text{ms} \quad (\mathbf{300\times \text{ faster!}}) Hit Cost=$0.0000vs.Miss Cost=$0.0025(100% cost savings!)\text{Hit Cost} = \$0.0000 \quad \text{vs.} \quad \text{Miss Cost} = \$0.0025 \quad (\mathbf{100\% \text{ cost savings!}})


In Simple English: Caching by Meaning, Not Exact Spelling (Semantic Cache)

Suppose User A asks: "Why is checkout crashing?" and User B asks 1 minute later: "Why did checkout-api crash?"

  • Exact Hash Cache (SHA-256): Misses! One different character breaks exact matching, so you pay LLM cost twice.
  • Semantic Prompt Cache: Embeds both questions into vectors and checks Cosine Similarity≥0.95\text{Cosine Similarity} \ge 0.95. When meaning matches (0.97 >= 0.95), return the saved answer in 6 ms at $0.0000! Remember from Week 12: ALWAYS scope cache lookups per tenant_id + role so Company B never reads Company A's cached secrets!