Week 7 • Day 5Week 7
Day 5:Week 7, Day 5: Inference Optimization — `KV-Cache` Mechanics, VRAM Math & Grouped-Query Attention
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 7, Day 5: Inference Optimization — KV-Cache Mechanics, VRAM Math & Grouped-Query Attention (GQA)
1. Why Naive Autoregressive Decoding is
When generating a 1,000-token response one token at a time, at step we only need the logits for the newest token :
Notice two crucial facts because of causal masking:
- Earlier Keys and Values never change when token is appended!
- If we recompute and from scratch at every step, generating tokens costs FLOPs!
The Solution: KV-Cache
By caching the past Key and Value tensors in GPU VRAM:
- At step , we only pass the single new token into the model!
- We compute , append to the cache, and compute in time instead of !
2. Exact KV-Cache GPU VRAM Formula (Staff Interview Classic)
For each token, at each layer, we store 2 tensors ( and ) across Key-Value heads of dimension :
Concrete Calculation: Llama-2-70B (MHA) vs Llama-3-70B (GQA)
Suppose layers, ( query heads, ), BF16 (2 bytes), batch size , context :
- Standard MHA (): $2 \times 80 \times 64 \times 128 \times 32768 \times 1 \times 2 = \mathbf{80.0\text{ GiB of VRAM!}}$
- Grouped-Query Attention (
GQAwith ,8:1ratio): $2 \times 80 \times 8 \times 128 \times 32768 \times 1 \times 2 = \mathbf{10.0\text{ GiB of VRAM! (8x reduction!)}}$
In Simple English: KV Cache = Not Re-Reading the Whole Book for Every New Word
Suppose you ask an LLM to write a 100-word essay:
- Without KV Cache: To write word #99, it re-reads words #1..98 and recomputes all their Key () and Value () vectors from scratch. Writing word #100 repeats everything AGAIN ( wasted work)!
- With KV Cache: Save the and vectors of words #1..99 inside GPU VRAM! Now generating word #100 computes ONLY the 1 new Query vector and looks up the saved history in a single pass!
- Catch (VRAM Formula): Every extra active user in a batch needs gigabytes of saved vectors. Grouped-Query Attention (
GQA) lets e.g. 4 Query heads share 1 Key/Value head — cutting KV-Cache VRAM usage by 4x–8x so servers handle far more concurrent users!