Day 5:Week 6, Day 5: Custom `Dataset`, `DataLoader`, Dynamic Padding & Attention Masks
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 6, Day 5: Custom Dataset, DataLoader, Dynamic Padding & Attention Masks
1. Why Batching Variable-Length Text is Non-Trivial
GPUs require rectangular tensors of shape (batch_size, seq_len) to execute parallel matrix multiplications. However, real-world texts have different lengths:
- Log 1:
"DB timeout"tokens:[14, 89] - Log 2:
"Payment gateway HTTP 502 error on checkout"tokens:[22, 41, 90, 105, 12, 77]
You cannot stack a length-2 tensor and a length-6 tensor into a 2D matrix without Padding!
2. Static Padding vs Dynamic Batch Padding (collate_fn)
The Anti-Pattern: Static Padding to max_context_length
If your model supports up to 2,048 tokens and you pad every short 10-token example to 2,048 using <PAD> = 0:
- You waste 99.5% of GPU memory and compute multiplying
<PAD>tokens!
The Production Pattern: Dynamic Padding inside collate_fn
Instead of padding inside Dataset.__getitem__, return raw variable-length sequences from Dataset, and pass a custom collate_fn to DataLoader that pads each mini-batch only to the longest sequence in that specific batch:
Batch Sequences before collate_fn:
Sample 0: [14, 89] (len 2)
Sample 1: [22, 41, 90, 105, 12, 77] (len 6 -> max in batch!)
Sample 2: [55, 19, 8] (len 3)
After Dynamic Padding (PAD_ID = 0) -> Shape (3, 6):
input_ids:
[[14, 89, 0, 0, 0, 0],
[22, 41, 90, 105, 12, 77],
[55, 19, 8, 0, 0, 0]]
attention_mask (1 = real token, 0 = pad token) -> Shape (3, 6):
[[1, 1, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1],
[1, 1, 1, 0, 0, 0]]
3. Why attention_mask and Masked Mean Pooling are Critical
Even when you set padding_idx=0 in nn.Embedding, if you average across all seq_len positions (embeddings.mean(dim=1)), a 2-token sentence padded to 6 tokens has its signal diluted by zeros!
Mathematical Formulation of Masked Mean Pooling
Given embeddings and binary mask :
By multiplying each token embedding by and dividing by the true unpadded token count , <PAD> tokens have zero mathematical effect on the pooled sentence representation!
In Simple English: Why Padding & Attention Masks Exist
Computers process batches of sentences together in rectangular grids — just like packing different-length sticks into a shipping box:
- Sentence 1 has 4 words:
"Server crashed right now" - Sentence 2 has 2 words:
"Help me"To match Sentence 1 in the same box, we pad it with blank filler tokens:"Help me <PAD> <PAD>". - What Is the Attention Mask (
[1, 1, 0, 0])? It tells the neural network:1= real word (read this!),0= empty packing foam (completely ignore this!) so blank padding never dilutes the sentence's average meaning!