Week 6 • Day 5Week 6

Day 5:Week 6, Day 5: Custom `Dataset`, `DataLoader`, Dynamic Padding & Attention Masks

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 6, Day 5: Custom Dataset, DataLoader, Dynamic Padding & Attention Masks


1. Why Batching Variable-Length Text is Non-Trivial

GPUs require rectangular tensors of shape (batch_size, seq_len) to execute parallel matrix multiplications. However, real-world texts have different lengths:

  • Log 1: "DB timeout" →2\rightarrow 2 tokens: [14, 89]
  • Log 2: "Payment gateway HTTP 502 error on checkout" →6\rightarrow 6 tokens: [22, 41, 90, 105, 12, 77]

You cannot stack a length-2 tensor and a length-6 tensor into a 2D matrix without Padding!


2. Static Padding vs Dynamic Batch Padding (collate_fn)

The Anti-Pattern: Static Padding to max_context_length

If your model supports up to 2,048 tokens and you pad every short 10-token example to 2,048 using <PAD> = 0:

  • You waste 99.5% of GPU memory and compute multiplying <PAD> tokens!

The Production Pattern: Dynamic Padding inside collate_fn

Instead of padding inside Dataset.__getitem__, return raw variable-length sequences from Dataset, and pass a custom collate_fn to DataLoader that pads each mini-batch only to the longest sequence in that specific batch:

Lbatch=max⁡i∈batchlen(xi)L_{\text{batch}} = \max_{i \in \text{batch}} \text{len}(x_i)

code
Batch Sequences before collate_fn: Sample 0: [14, 89] (len 2) Sample 1: [22, 41, 90, 105, 12, 77] (len 6 -> max in batch!) Sample 2: [55, 19, 8] (len 3) After Dynamic Padding (PAD_ID = 0) -> Shape (3, 6): input_ids: [[14, 89, 0, 0, 0, 0], [22, 41, 90, 105, 12, 77], [55, 19, 8, 0, 0, 0]] attention_mask (1 = real token, 0 = pad token) -> Shape (3, 6): [[1, 1, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1], [1, 1, 1, 0, 0, 0]]

3. Why attention_mask and Masked Mean Pooling are Critical

Even when you set padding_idx=0 in nn.Embedding, if you average across all seq_len positions (embeddings.mean(dim=1)), a 2-token sentence padded to 6 tokens has its signal diluted by 46=66.7%\frac{4}{6} = 66.7\% zeros!

Mathematical Formulation of Masked Mean Pooling

Given embeddings E∈RB×T×DE \in \mathbb{R}^{B \times T \times D} and binary mask M∈{0,1}B×TM \in \{0, 1\}^{B \times T}:

μb=∑t=1TMb,t⋅Eb,tmax⁡(∑t=1TMb,t,  1)\mu_b = \frac{\sum_{t=1}^{T} M_{b,t} \cdot E_{b,t}}{\max\left(\sum_{t=1}^{T} M_{b,t},\; 1\right)}

By multiplying each token embedding by Mb,t∈{0,1}M_{b,t} \in \{0, 1\} and dividing by the true unpadded token count ∑tMb,t\sum_t M_{b,t}, <PAD> tokens have zero mathematical effect on the pooled sentence representation!


In Simple English: Why Padding & Attention Masks Exist

Computers process batches of sentences together in rectangular grids — just like packing different-length sticks into a shipping box:

  • Sentence 1 has 4 words: "Server crashed right now"
  • Sentence 2 has 2 words: "Help me" →\rightarrow To match Sentence 1 in the same box, we pad it with blank filler tokens: "Help me <PAD> <PAD>".
  • What Is the Attention Mask ([1, 1, 0, 0])? It tells the neural network: 1 = real word (read this!), 0 = empty packing foam (completely ignore this!) so blank padding never dilutes the sentence's average meaning!