Week 17 • Day 3Week 17

Day 3:Multi-Turn Chat Memory, Sliding Windows & Background Compaction

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 3: Multi-Turn Chat Memory, Sliding Windows & Background Compaction

1. First-Principles Problem: The Quadratic Context Window Tax

In stateless LLM API interactions, "memory" is simulated by sending the entire conversation history on every single HTTP request.

If each turn adds ΔT=400\Delta T = 400 tokens:

  • Turn 1 input: $400$ tokens
  • Turn 2 input: $800$ tokens
  • Turn KK input: K⋅400K \cdot 400 tokens

Cumulative Input Tokens Across KK Turns:

Tokens(K)=∑k=1K400⋅k=400⋅K(K+1)2=O(K2)\text{Tokens}(K) = \sum_{k=1}^K 400 \cdot k = 400 \cdot \frac{K(K+1)}{2} = O(K^2) For K=25K = 25 turns: Tokens(25)=400⋅25⋅262=130,000 tokens billed!\text{Tokens}(25) = 400 \cdot \frac{25 \cdot 26}{2} = 130,000 \text{ tokens billed!} 90% of those tokens are spent re-reading early pleasantries or solved debugging steps that have zero relevance to the current turn.


2. Production Three-Tier Memory Architecture

Architecture Flow Diagram
Incoming Turn K->Store[Session Store (Redis / Postgres)
Buffer[Tier 1: Recent Verbatim Window (Last 4-6 Messages)
Compactor[Tier 2: Sliding Token Trimmer (Cap: 2,500 Tokens)
Summary[Tier 3: Asynchronous Background Compaction / Fact Store
Assembled[Assembled Prompt = System Prompt + Rolling Summary + Recent Verbatim Messages
  1. System Prompt (Pinned): Base persona and role constraints ($150\text{ tokens}$).
  2. Rolling Summary (Dynamic): A dense 1-paragraph synthesis of all historical turns older than the recent window ($100\text{–}200\text{ tokens}$).
  3. Recent Verbatim Window: The latest MM messages (e.g., last 4 turns) preserving exact code snippets and parameters ($800\text{ tokens}$).

Token Stability Result:

Input Tokens per Turn≤150+200+800=1,150 tokens (Flat O(1) scaling!)\text{Input Tokens per Turn} \le 150 + 200 + 800 = 1,150 \text{ tokens (Flat } O(1) \text{ scaling!)} Instead of $130,000 tokens over 25 turns, the bounded memory consumes only \28,750$ tokens—a 77.8% cost reduction with zero context window overflow!


In Simple English: Why Unbounded Chat History Blows Up Your Bill (K(K+1)2\frac{K(K+1)}{2})

Suppose every message adds 400 words, and you resend the entire chat history on every reply:

  • Turn 1 sends 400 tokens. Turn 2 sends 800. Turn 3 sends 1,200...
  • By Turn 25, you are paying for $400 + 800 + \dots + 10,000 = \mathbf{130,000\text{ tokens}}$ just for ONE conversation! Production Fix (Sliding Window + Compaction): Keep the last 4–6 exact turns verbatim, and summarize turns 1–19 into a compact 200-token sticky-note summary in the background — keeping input cost flat (~1,150 tokens/turn) forever!