Week 17 • Day 3Week 17
Day 3:Multi-Turn Chat Memory, Sliding Windows & Background Compaction
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 3: Multi-Turn Chat Memory, Sliding Windows & Background Compaction
1. First-Principles Problem: The Quadratic Context Window Tax
In stateless LLM API interactions, "memory" is simulated by sending the entire conversation history on every single HTTP request.
If each turn adds tokens:
- Turn 1 input: $400$ tokens
- Turn 2 input: $800$ tokens
- Turn input: tokens
Cumulative Input Tokens Across Turns:
For turns: 90% of those tokens are spent re-reading early pleasantries or solved debugging steps that have zero relevance to the current turn.
2. Production Three-Tier Memory Architecture
Architecture Flow Diagram
Incoming Turn K->Store[Session Store (Redis / Postgres)
Buffer[Tier 1: Recent Verbatim Window (Last 4-6 Messages)
Compactor[Tier 2: Sliding Token Trimmer (Cap: 2,500 Tokens)
Summary[Tier 3: Asynchronous Background Compaction / Fact Store
Assembled[Assembled Prompt = System Prompt + Rolling Summary + Recent Verbatim Messages
- System Prompt (Pinned): Base persona and role constraints ($150\text{ tokens}$).
- Rolling Summary (Dynamic): A dense 1-paragraph synthesis of all historical turns older than the recent window ($100\text{–}200\text{ tokens}$).
- Recent Verbatim Window: The latest messages (e.g., last 4 turns) preserving exact code snippets and parameters ($800\text{ tokens}$).
Token Stability Result:
Instead of $130,000 tokens over 25 turns, the bounded memory consumes only \28,750$ tokens—a 77.8% cost reduction with zero context window overflow!
In Simple English: Why Unbounded Chat History Blows Up Your Bill ()
Suppose every message adds 400 words, and you resend the entire chat history on every reply:
- Turn 1 sends
400tokens. Turn 2 sends800. Turn 3 sends1,200... - By Turn 25, you are paying for $400 + 800 + \dots + 10,000 = \mathbf{130,000\text{ tokens}}$ just for ONE conversation!
Production Fix (
Sliding Window + Compaction): Keep the last 4–6 exact turns verbatim, and summarize turns 1–19 into a compact 200-token sticky-note summary in the background — keeping input cost flat (~1,150 tokens/turn) forever!