Day 6:Week 6, Day 6: Overfitting, Inverted Dropout, Gradient Clipping & Mixed Precision
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 6, Day 6: Overfitting, Inverted Dropout, Gradient Clipping & Mixed Precision (FP16 vs BF16)
1. Inverted Dropout (nn.Dropout) — Why model.eval() Matters
During training, a deep neural network can memorize co-adapted feature pairs in the training data (Overfitting). Dropout randomly zeroes out each activation with probability during training.
Why PyTorch Uses "Inverted Dropout"
If you zero out a fraction of neurons during training, the expected sum of activations shrinks by . To keep the expected magnitude identical between training and inference without having to multiply weights at inference time, PyTorch scales the surviving activations during model.train() by :
Verify the expectation during training:
[!CAUTION] Production Bug Alert: If you forget to call
model.eval()before running production inference,nn.Dropoutstays active—making your model output random, non-deterministic predictions on every API request!
2. Gradient Clipping (torch.nn.utils.clip_grad_norm_)
Even in Transformers and LSTMs, a bad mini-batch or steep loss cliff can produce a massive gradient norm , launching weights into NaN territory in a single optimizer step.
Global Norm Gradient Clipping rescales the entire parameter gradient vector if its global norm exceeds max_norm (, typically 1.0 in LLM training):
Notice that this preserves the exact direction of the gradient vector while bounding its step size!
3. Floating-Point Formats in AI: FP32 vs FP16 vs BF16 vs INT8
Every AI Engineer training or serving LLMs must know the exact bit layouts of GPU data types:
FP32 (32-bit Full Precision, 4 bytes):
[1 Sign bit] [8 Exponent bits] [23 Mantissa (Fraction) bits]
Max Value: ~3.4 x 10^38
FP16 (16-bit IEEE Half Precision, 2 bytes):
[1 Sign bit] [5 Exponent bits] [10 Mantissa bits]
Max Value: 65,504 <-- DANGER! Gradients/activations > 65,504 overflow to Inf/NaN!
BF16 (16-bit Brain Floating Point - Google/NVIDIA Ampere+, 2 bytes):
[1 Sign bit] [8 Exponent bits] [7 Mantissa bits]
Max Value: ~3.4 x 10^38 <-- SAME dynamic range as FP32! No loss scaling needed!
Why BF16 (bfloat16) Revolutionized LLM Training
In FP16 (5 exponent bits), any activation or unscaled gradient above 65,504 overflows to Inf, and small gradients underflow to 0.0 (requiring GradScaler).
BF16 trades 3 mantissa bits for 8 exponent bits—giving BF16 the exact same dynamic range as FP32 at half the memory (2 bytes/param)!
In Simple English: Stopping Memorization (Dropout) & Exploding Updates (Clipping)
- Overfitting Analogy: A student memorizes exact practice exam answers ("Question 4 = B") instead of learning concepts — then scores 100% in practice and 40% on the real test!
- Inverted Dropout: During training, randomly turn off e.g. 20% () of neurons on every batch (like forcing a soccer team to win even when random players sit out!). Each surviving neuron scales up by (
1 / 0.8 = 1.25x) so average brightness stays identical at test time when everyone plays! - Gradient Clipping: If a corrupted training batch suddenly causes a massive earthquake gradient (), clipping shrinks its length safely down to
1.0without changing its direction — preventing NaN crashes!