Day 2:Week 7, Day 2: Multi-Head Attention (`MHA`) & Autoregressive Causal Masking
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 7, Day 2: Multi-Head Attention (MHA) & Autoregressive Causal Masking
1. Why Single-Head Attention is Not Enough
If a token only has a single Query vector , its softmax distribution over the sequence tends to gets dominated by the single strongest relationship (e.g., syntactic subject-verb agreement). However, a token in a sentence simultaneously needs to track multiple independent relationships:
- Head 1: Coreference resolution ("it" "the databaseserver")
- Head 2: Syntactic verb-object binding
- Head 3: Adjacent n-gram / local context
- Head 4: Error code / key-value pairing
Mathematical Formulation of Multi-Head Attention (MHA)
Instead of performing one attention of dimension , we split into parallel heads, each of dimension :
[!TIP] Total Parameter Count Equivalence: Because , packing all heads into a single matrix and reshaping
(B, T, d_model) -> (B, h, T, d_k)has the exact same parameter count and FLOPs as single-head attention of dimension !
2. Autoregressive Causal Masking: Preventing Future Data Leakage
When training a Decoder-only LLM (GPT, Llama, Gemini) to predict the next token given , token must NEVER be allowed to attend to future tokens !
How do we enforce this for all positions in parallel during a single forward pass? We apply a Lower-Triangular Causal Mask where if and $0j > i$:
Because , after applying , every upper-triangular entry becomes strictly 0.0, guaranteeing zero information leakage from the future!
Causal Attention Weights Matrix (T = 4):
Key 0 Key 1 Key 2 Key 3
Query 0 [ 1.00, 0.00, 0.00, 0.00 ] <-- Token 0 only sees Token 0
Query 1 [ 0.42, 0.58, 0.00, 0.00 ] <-- Token 1 sees Tokens 0..1
Query 2 [ 0.20, 0.35, 0.45, 0.00 ] <-- Token 2 sees Tokens 0..2
Query 3 [ 0.10, 0.25, 0.30, 0.35 ] <-- Token 3 sees Tokens 0..3
In Simple English: Multi-Head Attention & Stopping LLMs From Cheating (Causal Mask)
- Why Multiple Heads (
MHA)? In the sentence "The tired dog refused to cross the busy street because it was scared", Head 1 learns grammar ("tired" describes "dog"), Head 2 solves pronoun links ("it" refers to "dog"), and Head 3 watches location ("street"). Having 8–32 parallel heads lets the model wear 8 different pairs of glasses at once! - Causal Masking (Blocking Future Peeking): While training an LLM to predict the next word in "The cat sat on the mat", token #3 ("sat") is forbidden from looking ahead at words #4–6 ("on the mat"). Setting future cells to turns their Softmax probability to strictly
0.0— like covering the right side of a page with a sheet of paper while studying!