Week 22 • Day 1Week 22

Day 1:The Physics & Math of LLM Inference (VRAM & Bandwidth Calculators)

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 1: The Physics & Math of LLM Inference (VRAM & Bandwidth Calculators)

1. Executive Summary: The Memory Wall

Senior AI Engineers do not guess how many GPUs are needed to serve a model—they calculate the exact memory footprint and token throughput down to the byte.

The central insight of LLM inference is the Roofline Model:

  • Prefill (Prompt Processing) is Compute-Bound: Tokens are processed in parallel via General Matrix Multiply (GEMM). Throughput depends on GPU Tensor Core TFLOPs.
  • Decoding (Token Generation) is Memory-Bandwidth Bound: Tokens are generated autoregressively one-by-one via General Matrix-Vector Multiply (GEMV). Throughput depends strictly on High Bandwidth Memory (HBM) bandwidth (GB/s\text{GB/s}).

2. Mathematical VRAM Formulations

1. Static Model Weights

Memoryweights=P×b8×109 GB\text{Memory}_{\text{weights}} = \frac{P \times b}{8 \times 10^9}\text{ GB} where PP is parameter count and bb is bits per parameter:

  • FP16 / BF16: b=16b = 16 ($2\text{ bytes per param}$).
  • INT8 / FP8: b=8b = 8 ($1\text{ byte per param}$).
  • INT4 / AWQ / NF4: b=4b = 4 ($0.5\text{ bytes per param}$).

For an 8-billion parameter model:

  • 16-bit: $8 \times 2 = 16\text{ GB}$.
  • 4-bit: $8 \times 0.5 = 4\text{ GB}$.

2. The Dynamic KV Cache: First Principles

In multi-head attention, past key and value vectors must be retained in GPU memory to compute attention for newly generated tokens without recomputing the entire sequence: KV Cache Size per Token=2×nlayers×nkv_heads×dhead×bytes_per_element\text{KV Cache Size per Token} = 2 \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \times \text{bytes\_per\_element} where the factor of $2$ accounts for both Keys and Values.

Multi-Head Attention (MHA) vs. Grouped-Query Attention (GQA)

In traditional MHA (e.g., LLaMA-1), the number of KV heads equals the number of query heads (nkv_heads=nquery_heads=32n_{\text{kv\_heads}} = n_{\text{query\_heads}} = 32). Modern models (LLaMA-3, Qwen-2.5, Mistral) employ Grouped-Query Attention (GQA), sharing 1 KV head across 4 or 8 query heads:

  • LLaMA-3.1-8B: nlayers=32,nkv_heads=8,dhead=128n_{\text{layers}} = 32, n_{\text{kv\_heads}} = 8, d_{\text{head}} = 128. KVtoken=2×32×8×128×2 bytes=131,072 bytes≈128 KB per token\text{KV}_{\text{token}} = 2 \times 32 \times 8 \times 128 \times 2\text{ bytes} = 131,072\text{ bytes} \approx \mathbf{128\text{ KB per token}} Across a batch of B=16B = 16 concurrent requests with context length L=4,096L = 4,096 tokens: Total KV Cache=16×4096×128 KB≈8.39 GB of VRAM!\text{Total KV Cache} = 16 \times 4096 \times 128\text{ KB} \approx \mathbf{8.39\text{ GB of VRAM!}}
Architecture Flow Diagram
Multi-Head Attention (MHA)<br/>32 Query Heads : 32 KV Heads<br/>KV Cache: 512 KB/token
Grouped-Query Attention (GQA)<br/>32 Query Heads : 8 KV Heads<br/>KV Cache: 128 KB/token (4x Savings!)
Multi-Query Attention (MQA)<br/>32 Query Heads : 1 KV Head<br/>KV Cache: 16 KB/token (32x Savings)

3. Decoding Throughput: The Memory Bandwidth Bottleneck

During the autoregressive decode phase, to generate a single new token for a single request, the GPU must transfer all model parameters from HBM into the on-chip registers.

Latency per Token (Single Stream)=Model Size in BytesGPU Memory Bandwidth (Bytes/s)\text{Latency per Token (Single Stream)} = \frac{\text{Model Size in Bytes}}{\text{GPU Memory Bandwidth (Bytes/s)}} Maximum Theoretical Decode Speed=GPU Memory Bandwidth (GB/s)Model Size (GB)\text{Maximum Theoretical Decode Speed} = \frac{\text{GPU Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}

Industry Hardware Matrix

HardwareMemory CapacityHBM BandwidthMax Decode (8B FP16 = 16GB)Max Decode (8B 4-bit = 4GB)
NVIDIA H100 SXM580 GB HBM33,350 GB/s$209\text{ tokens/s}$837 tokens/s\mathbf{837\text{ tokens/s}}
NVIDIA A100 SXM480 GB HBM2e2,039 GB/s$127\text{ tokens/s}$509 tokens/s\mathbf{509\text{ tokens/s}}
NVIDIA RTX 409024 GB GDDR6X1,008 GB/s$63\text{ tokens/s}$252 tokens/s\mathbf{252\text{ tokens/s}}
NVIDIA L4 (Cloud)24 GB GDDR6300 GB/s$18.7\text{ tokens/s}$75 tokens/s\mathbf{75\text{ tokens/s}}
Apple M3 Max128 GB Unified400 GB/s$25\text{ tokens/s}$100 tokens/s\mathbf{100\text{ tokens/s}}

Key Takeaway: A 4-bit quantized model generates tokens 4 times faster on the exact same GPU because it requires transferring only 4 GB over the memory bus per step instead of 16 GB!


In Simple English: Memory Wall Arithmetic (How Fast Can Any GPU Generate Words?)

Every word generated during LLM decoding forces the GPU to stream ALL model weights from VRAM into compute cores once!

  • Weight Memory Rule of Thumb: Divide parameter count by 2 for FP16/BF16 (70B parameters ×2\times 2 bytes ≈140\approx 140 GB), or divide by 2 again (factor 4 total -> divide by 2) for 4-bit INT4/AWQ (70B ×0.5\times 0.5 bytes ≈35\approx 35 GB!).
  • Decode Speed Ceiling (tokens/sec): An NVIDIA A100 has ~2,000 GB/s VRAM bandwidth. Running a 70 GB model yields at most $2,000 / 70 \approx \mathbf{28\text{ tokens/sec}}$ per single user — no matter how fast its matrix cores are, memory bandwidth sets the speed limit!