Day 1:The Physics & Math of LLM Inference (VRAM & Bandwidth Calculators)
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 1: The Physics & Math of LLM Inference (VRAM & Bandwidth Calculators)
1. Executive Summary: The Memory Wall
Senior AI Engineers do not guess how many GPUs are needed to serve a model—they calculate the exact memory footprint and token throughput down to the byte.
The central insight of LLM inference is the Roofline Model:
- Prefill (Prompt Processing) is Compute-Bound: Tokens are processed in parallel via General Matrix Multiply (GEMM). Throughput depends on GPU Tensor Core TFLOPs.
- Decoding (Token Generation) is Memory-Bandwidth Bound: Tokens are generated autoregressively one-by-one via General Matrix-Vector Multiply (GEMV). Throughput depends strictly on High Bandwidth Memory (HBM) bandwidth ().
2. Mathematical VRAM Formulations
1. Static Model Weights
where is parameter count and is bits per parameter:
- FP16 / BF16: ($2\text{ bytes per param}$).
- INT8 / FP8: ($1\text{ byte per param}$).
- INT4 / AWQ / NF4: ($0.5\text{ bytes per param}$).
For an 8-billion parameter model:
- 16-bit: $8 \times 2 = 16\text{ GB}$.
- 4-bit: $8 \times 0.5 = 4\text{ GB}$.
2. The Dynamic KV Cache: First Principles
In multi-head attention, past key and value vectors must be retained in GPU memory to compute attention for newly generated tokens without recomputing the entire sequence: where the factor of $2$ accounts for both Keys and Values.
Multi-Head Attention (MHA) vs. Grouped-Query Attention (GQA)
In traditional MHA (e.g., LLaMA-1), the number of KV heads equals the number of query heads (). Modern models (LLaMA-3, Qwen-2.5, Mistral) employ Grouped-Query Attention (GQA), sharing 1 KV head across 4 or 8 query heads:
- LLaMA-3.1-8B: . Across a batch of concurrent requests with context length tokens:
3. Decoding Throughput: The Memory Bandwidth Bottleneck
During the autoregressive decode phase, to generate a single new token for a single request, the GPU must transfer all model parameters from HBM into the on-chip registers.
Industry Hardware Matrix
| Hardware | Memory Capacity | HBM Bandwidth | Max Decode (8B FP16 = 16GB) | Max Decode (8B 4-bit = 4GB) |
|---|---|---|---|---|
| NVIDIA H100 SXM5 | 80 GB HBM3 | 3,350 GB/s | $209\text{ tokens/s}$ | |
| NVIDIA A100 SXM4 | 80 GB HBM2e | 2,039 GB/s | $127\text{ tokens/s}$ | |
| NVIDIA RTX 4090 | 24 GB GDDR6X | 1,008 GB/s | $63\text{ tokens/s}$ | |
| NVIDIA L4 (Cloud) | 24 GB GDDR6 | 300 GB/s | $18.7\text{ tokens/s}$ | |
| Apple M3 Max | 128 GB Unified | 400 GB/s | $25\text{ tokens/s}$ |
Key Takeaway: A 4-bit quantized model generates tokens 4 times faster on the exact same GPU because it requires transferring only 4 GB over the memory bus per step instead of 16 GB!
In Simple English: Memory Wall Arithmetic (How Fast Can Any GPU Generate Words?)
Every word generated during LLM decoding forces the GPU to stream ALL model weights from VRAM into compute cores once!
- Weight Memory Rule of Thumb: Divide parameter count by 2 for FP16/BF16 (70B parameters bytes GB), or divide by 2 again (factor 4 total -> divide by 2) for 4-bit INT4/AWQ (70B bytes GB!).
- Decode Speed Ceiling (
tokens/sec): An NVIDIA A100 has ~2,000 GB/s VRAM bandwidth. Running a 70 GB model yields at most $2,000 / 70 \approx \mathbf{28\text{ tokens/sec}}$ per single user — no matter how fast its matrix cores are, memory bandwidth sets the speed limit!