Week 21 • Day 4Week 21

Day 4:QLoRA & 4-bit NormalFloat (`NF4`) Quantization Internals

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 4: QLoRA & 4-bit NormalFloat (NF4) Quantization Internals

1. Executive Summary & The Memory Breakthrough

In 2023, Tim Dettmers et al. published QLoRA: Efficient Finetuning of Quantized LLMs, unlocking the ability to fine-tune a 65-billion parameter model on a single 48GB GPU (or a 7B/8B model on a single 16GB consumer GPU) with zero performance degradation compared to 16-bit full fine-tuning.

QLoRA achieved this through three architectural innovations:

  1. 4-Bit NormalFloat (NF4): An information-theoretically optimal data type for normally distributed neural network weights.
  2. Double Quantization (DQ): Quantizing the quantization constants (scales) themselves to save $0.37$ bits per parameter.
  3. Paged Optimizers: Utilizing CUDA Unified Memory to automatically page optimizer states between GPU VRAM and CPU RAM during allocation spikes.

2. Why Uniform Integer Quantization (INT4) Fails for LLMs

Standard 4-bit integer quantization (INT4) distributes $2^4 = 16$ quantization bins uniformly across the dynamic range: qi=round(wscale)∈{−8,−7,…,0,…,7}q_i = \text{round}\left( \frac{w}{\text{scale}} \right) \in \{-8, -7, \dots, 0, \dots, 7\}

The Gaussian Mismatch

Pre-trained neural network weights are not uniformly distributed. Following standard weight initialization and regularization, LLM weights follow a zero-mean normal distribution: W∼N(0,σ2)W \sim \mathcal{N}(0, \sigma^2)

Architecture Flow Diagram
Linear Bins: Equal Spacing [-3σ to +3σ]
Wastes bits on empty tails (±3σ)
Coarse resolution at high-density center (around 0)
Quantile Bins: Variable Spacing
Dense bins around 0 where 80% of weights lie
Sparse bins at tails to capture outlier magnitudes

Uniform quantization wastes most of its 16 bins on near-empty tail regions, while providing inadequate resolution in the dense cluster around zero. This leads to high quantization noise and catastrophic perplexity degradation.


3. Mathematical Construction of NormalFloat4 (NF4)

The NF4 data type constructs $2^kbins( bins (k = 4,so16bins)suchthateachbinhas∗∗equalprobabilitymass∗∗underthestandardnormaldistribution, so 16 bins) such that each bin has **equal probability mass** under the standard normal distribution \mathcal{N}(0, 1)$.

Let QX(⋅)Q_X(\cdot) be the quantile function (inverse cumulative distribution function Φ−1\Phi^{-1}) of the standard normal distribution N(0,1)\mathcal{N}(0, 1). The continuous theoretical quantiles are: qi=Φ−1(i2k+12k+1)for i∈{0,1,…,15}q_i = \Phi^{-1}\left( \frac{i}{2^k} + \frac{1}{2^{k+1}} \right) \quad \text{for } i \in \{0, 1, \dots, 15\}

To ensure an exact representation of zero (essential for padding and zero-valued gradients), the distribution is split into positive and negative halves, normalized to the range [−1.0,1.0][-1.0, 1.0]. The resulting exact 16-point NF4 codebook values are:

-1.0, & -0.6961928, & -0.5250731, & -0.3949175, \\ -0.2844414, & -0.1847734, & -0.0910500, & 0.0, \\ 0.0795803, & 0.1609302, & 0.2461123, & 0.3379153, \\ 0.4407098, & 0.5626170, & 0.7229568, & 1.0 \end{bmatrix}$$ --- ## 4. Double Quantization (DQ) Mathematics In standard block-wise quantization (with block size $B_1 = 64$), each block of 64 FP16 weights requires a 32-bit floating point scale factor $c_1$: $$\text{Memory Overhead} = \frac{32\text{ bits}}{64} = 0.5\text{ bits per parameter}$$ Across a 65-billion parameter model, this scale factor overhead consumes: $$65 \times 10^9 \times 0.5\text{ bits} \approx 4.06\text{ GB of VRAM!}$$ ### Double Quantization Algorithm Double Quantization treats the primary scale factors $c_1$ as inputs to a second quantization step with block size $B_2 = 256$, quantizing $c_1$ from FP32 into FP8 (8 bits): $$\text{Memory Overhead}_{\text{DQ}} = \frac{8\text{ bits}}{B_1} + \frac{32\text{ bits}}{B_1 \times B_2} = \frac{8}{64} + \frac{32}{64 \times 256} = 0.125 + 0.00195 \approx \mathbf{0.127\text{ bits per param}}$$ **Net Memory Savings**: \$0.5 - 0.127 = 0.373\text{ bits/param}$, saving $\approx 3.03\text{ GB}$ of VRAM. --- ## 5. Dequantization Forward Pass in QLoRA During the forward pass of a fine-tuning step: 1. Base weights $W_0^{\text{NF4}}$ are stored in 4-bit memory. 2. Only when an activation tensor $x$ arrives at the layer, a block of 4-bit weights is dequantized into BF16 in GPU registers/cache: $$\tilde{W} = c_1 \cdot \mathcal{C}_{\text{NF4}}[W^{\text{NF4}}]$$ 3. Compute forward activation: $$h = \tilde{W} x + \frac{\alpha}{r} B A x$$ 4. Dequantized BF16 weights are immediately evicted from cache, preserving 4-bit VRAM footprint. --- ## In Simple English: Fitting a 70B Model Onto One Consumer GPU With `QLoRA` (`NF4`) - **Why Uniform Rounding (`INT4`) Hurts AI Weights:** Almost 95% of LLM weights cluster near zero in a bell curve (**Gaussian distribution**). Evenly spaced ruler ticks waste slots on extreme numbers like `-1.0` and `+1.0` while jamming thousands of delicate middle weights into the same bucket! - **NormalFloat4 (`NF4`) Custom Ticks:** Places **more ruler ticks tightly packed near `0.0`** where weights actually live, and fewer ticks at the tails — keeping bell-curve accuracy sharp at just **4 bits (`0.5 bytes`) per weight**! - **Double Quantization:** Compresses even the quantization ruler numbers themselves — saving ~4 GB on a 65B–70B model so LoRA training fits comfortably inside a single GPU!