Week 22 • Day 4Week 22

Day 4:GPU Quantization Formats Compared — AWQ, GPTQ, GGUF, FP8

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 4: GPU Quantization Formats Compared — AWQ, GPTQ, GGUF, FP8

1. Executive Summary: The Quantization Zoo

Choosing the wrong quantization format for production LLM serving degrades accuracy or leaves half your GPU throughput on the table:

  • Deploying a GGUF model on an NVIDIA A100 cluster wastes high-bandwidth tensor cores.
  • Running unquantized FP16 wastes 50% to 75% of memory bandwidth.
  • Using uniform INT4 without activation protection destroys complex reasoning.

A Principal AI Engineer knows the mathematical trade-offs between AWQ, GPTQ, GGUF, and FP8.


2. Mathematical Breakdown of Quantization Standards

1. Activation-Aware Weight Quantization (AWQ)

Lin et al. (MIT, MLSys 2024)

  • Core Insight: Weights are not created equal. Only $0.1% to \1%$ of weights are critical to model perplexity.
  • Rather than protecting weights with large magnitudes, AWQ protects weights connected to salient activation channels: s=sX=max⁡(∣X∣)s = \mathbf{s}_X = \max(|X|)
  • AWQ multiplies salient weight channels by a per-channel scaling factor s>1s > 1 while dividing incoming activations by ss: W′=W⋅diag⁡(s),X′=diag⁡(s)−1⋅XW' = W \cdot \operatorname{diag}(s), \quad X' = \operatorname{diag}(s)^{-1} \cdot X Mathematically: W′X′=(W⋅s)(s−1⋅X)=WXW' X' = (W \cdot s) (s^{-1} \cdot X) = W X (Exact mathematical equivalence!).
  • Production Verdict: The Gold Standard for GPU Serving (vLLM / TensorRT-LLM) on NVIDIA Ampere/Ada/Hopper GPUs.

2. GPTQ (Accurate Post-Training Quantization)

Frantar et al. (IST Austria, ICLR 2023)

  • Core Insight: Uses second-order Taylor expansion (Hessian matrix) to compensate for quantization error: arg⁡min⁡W^Tr⁡((W−W^)TH(W−W^))where H=2XXT\arg\min_{\hat{W}} \operatorname{Tr}\left( (W - \hat{W})^T H (W - \hat{W}) \right) \quad \text{where } H = 2 X X^T
  • Quantizes columns greedily one-by-one: when column qq is rounded to INT4, the error is distributed to all remaining unquantized columns j>qj > q using the inverse Hessian H−1H^{-1}.
  • Production Verdict: Good for batch-1 edge GPUs, but slightly higher calibration perplexity than AWQ at low bitwidths.

3. GGUF (k-quants: Q4_K_M, Q5_K_M)

Georgi Gerganov (llama.cpp)

  • Core Insight: Block-level mixed precision tailored for CPU and Apple Metal instruction sets (NEON / AVX-512 / Metal Shading Language).
  • Groups weights into 256-element super-blocks composed of 32-element sub-blocks, applying non-linear scaling and preserving critical attention down-projections in higher bits.
  • Production Verdict: The Undisputed King of CPU & Apple Silicon Serving (Ollama / llama.cpp).

4. FP8 (8-bit Floating Point: E4M3 vs. E5M2)

NVIDIA, ARM, Intel Standard

  • Supported in native hardware on NVIDIA Ada Lovelace (L4, RTX 4090) and Hopper (H100) Tensor Cores.
  • No dequantization overhead! Unlike INT4 (which must be dequantized back to FP16 before GEMM execution), FP8 is computed natively inside hardware Tensor Cores.
Architecture Flow Diagram
FP8 E4M3<br/>1 Sign | 4 Exponent | 3 Mantissa<br/>Dynamic Range: [-448, 448]<br/>Higher Precision (1/8 step)<br/><b>Used for: Weights & Activations</b>
FP8 E5M2<br/>1 Sign | 5 Exponent | 2 Mantissa<br/>Dynamic Range: [-57344, 57344]<br/>Identical range to FP16<br/><b>Used for: Gradients & KV Cache</b>

3. Production Decision Matrix

Architecture Flow Diagram
Deploying Open Foundation Model->HW{Target Hardware Environment?}
-- Apple Silicon (Mac) OR x86/ARM CPU->GGUF[Select GGUF (Q4_K_M / Q5_K_M)<br/>via Ollama / llama.cpp
-- NVIDIA Hopper (H100) OR Ada (L4)->FP8_DEC{High Batch Concurrency OR Native Tensor Cores?}
-- NVIDIA Ampere (A100 / A10G) OR Consumer (RTX 3090/4090)->AWQ[Select AWQ 4-bit<br/>via vLLM / TensorRT-LLM
-- Yes (Maximum Tensor Core FLOPs)->FP8[Select FP8 (E4M3)<br/>Native Hardware Serving
-- Memory-Constrained 70B Model->AWQ