Week 22 • Day 4Week 22
Day 4:GPU Quantization Formats Compared — AWQ, GPTQ, GGUF, FP8
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 4: GPU Quantization Formats Compared — AWQ, GPTQ, GGUF, FP8
1. Executive Summary: The Quantization Zoo
Choosing the wrong quantization format for production LLM serving degrades accuracy or leaves half your GPU throughput on the table:
- Deploying a GGUF model on an NVIDIA A100 cluster wastes high-bandwidth tensor cores.
- Running unquantized FP16 wastes 50% to 75% of memory bandwidth.
- Using uniform INT4 without activation protection destroys complex reasoning.
A Principal AI Engineer knows the mathematical trade-offs between AWQ, GPTQ, GGUF, and FP8.
2. Mathematical Breakdown of Quantization Standards
1. Activation-Aware Weight Quantization (AWQ)
Lin et al. (MIT, MLSys 2024)
- Core Insight: Weights are not created equal. Only $0.1% to \1%$ of weights are critical to model perplexity.
- Rather than protecting weights with large magnitudes, AWQ protects weights connected to salient activation channels:
- AWQ multiplies salient weight channels by a per-channel scaling factor while dividing incoming activations by : Mathematically: (Exact mathematical equivalence!).
- Production Verdict: The Gold Standard for GPU Serving (vLLM / TensorRT-LLM) on NVIDIA Ampere/Ada/Hopper GPUs.
2. GPTQ (Accurate Post-Training Quantization)
Frantar et al. (IST Austria, ICLR 2023)
- Core Insight: Uses second-order Taylor expansion (Hessian matrix) to compensate for quantization error:
- Quantizes columns greedily one-by-one: when column is rounded to INT4, the error is distributed to all remaining unquantized columns using the inverse Hessian .
- Production Verdict: Good for batch-1 edge GPUs, but slightly higher calibration perplexity than AWQ at low bitwidths.
3. GGUF (k-quants: Q4_K_M, Q5_K_M)
Georgi Gerganov (llama.cpp)
- Core Insight: Block-level mixed precision tailored for CPU and Apple Metal instruction sets (NEON / AVX-512 / Metal Shading Language).
- Groups weights into 256-element super-blocks composed of 32-element sub-blocks, applying non-linear scaling and preserving critical attention down-projections in higher bits.
- Production Verdict: The Undisputed King of CPU & Apple Silicon Serving (Ollama /
llama.cpp).
4. FP8 (8-bit Floating Point: E4M3 vs. E5M2)
NVIDIA, ARM, Intel Standard
- Supported in native hardware on NVIDIA Ada Lovelace (L4, RTX 4090) and Hopper (H100) Tensor Cores.
- No dequantization overhead! Unlike INT4 (which must be dequantized back to FP16 before GEMM execution), FP8 is computed natively inside hardware Tensor Cores.
Architecture Flow Diagram
FP8 E4M3<br/>1 Sign | 4 Exponent | 3 Mantissa<br/>Dynamic Range: [-448, 448]<br/>Higher Precision (1/8 step)<br/><b>Used for: Weights & Activations</b>
FP8 E5M2<br/>1 Sign | 5 Exponent | 2 Mantissa<br/>Dynamic Range: [-57344, 57344]<br/>Identical range to FP16<br/><b>Used for: Gradients & KV Cache</b>
3. Production Decision Matrix
Architecture Flow Diagram
Deploying Open Foundation Model->HW{Target Hardware Environment?}
-- Apple Silicon (Mac) OR x86/ARM CPU->GGUF[Select GGUF (Q4_K_M / Q5_K_M)<br/>via Ollama / llama.cpp
-- NVIDIA Hopper (H100) OR Ada (L4)->FP8_DEC{High Batch Concurrency OR Native Tensor Cores?}
-- NVIDIA Ampere (A100 / A10G) OR Consumer (RTX 3090/4090)->AWQ[Select AWQ 4-bit<br/>via vLLM / TensorRT-LLM
-- Yes (Maximum Tensor Core FLOPs)->FP8[Select FP8 (E4M3)<br/>Native Hardware Serving
-- Memory-Constrained 70B Model->AWQ