Week 7 • Day 2Week 7

Day 2:Week 7, Day 2: Multi-Head Attention (`MHA`) & Autoregressive Causal Masking

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 7, Day 2: Multi-Head Attention (MHA) & Autoregressive Causal Masking


1. Why Single-Head Attention is Not Enough

If a token only has a single Query vector qi∈Rdmodelq_i \in \mathbb{R}^{d_{\text{model}}}, its softmax distribution over the sequence tends to gets dominated by the single strongest relationship (e.g., syntactic subject-verb agreement). However, a token in a sentence simultaneously needs to track multiple independent relationships:

  • Head 1: Coreference resolution ("it" →\rightarrow "the databaseserver")
  • Head 2: Syntactic verb-object binding
  • Head 3: Adjacent n-gram / local context
  • Head 4: Error code / key-value pairing

Mathematical Formulation of Multi-Head Attention (MHA)

Instead of performing one attention of dimension dmodeld_{\text{model}}, we split dmodeld_{\text{model}} into hh parallel heads, each of dimension dk=dmodel/hd_k = d_{\text{model}} / h:

MultiHead(Q,K,V)=Concat(head1,…,headh)WO\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W_O where headi=Attention(XWQ(i),XWK(i),XWV(i))\text{where } \text{head}_i = \text{Attention}(X W_Q^{(i)}, X W_K^{(i)}, X W_V^{(i)})

[!TIP] Total Parameter Count Equivalence: Because h×dk=dmodelh \times d_k = d_{\text{model}}, packing all hh heads into a single matrix WQ∈Rdmodel×dmodelW_Q \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}} and reshaping (B, T, d_model) -> (B, h, T, d_k) has the exact same parameter count and FLOPs as single-head attention of dimension dmodeld_{\text{model}}!


2. Autoregressive Causal Masking: Preventing Future Data Leakage

When training a Decoder-only LLM (GPT, Llama, Gemini) to predict the next token xt+1x_{t+1} given x1,…,xtx_1, \dots, x_t, token tt must NEVER be allowed to attend to future tokens t+1,…,Tt+1, \dots, T!

How do we enforce this for all TT positions in parallel during a single forward pass? We apply a Lower-Triangular Causal Mask M∈{0,1}T×TM \in \{0, 1\}^{T \times T} where Mi,j=1M_{i,j} = 1 if j≤ij \le i and $0ififj > i$:

Si,jmasked={qi⋅kjdk,if j≤i−∞,if j>i (future token!)S_{i,j}^{\text{masked}} = \begin{cases} \frac{q_i \cdot k_j}{\sqrt{d_k}}, & \text{if } j \le i \\ -\infty, & \text{if } j > i \text{ (future token!)} \end{cases}

Because e−∞=0e^{-\infty} = 0, after applying softmax\text{softmax}, every upper-triangular entry (j>i)(j > i) becomes strictly 0.0, guaranteeing zero information leakage from the future!

code
Causal Attention Weights Matrix (T = 4): Key 0 Key 1 Key 2 Key 3 Query 0 [ 1.00, 0.00, 0.00, 0.00 ] <-- Token 0 only sees Token 0 Query 1 [ 0.42, 0.58, 0.00, 0.00 ] <-- Token 1 sees Tokens 0..1 Query 2 [ 0.20, 0.35, 0.45, 0.00 ] <-- Token 2 sees Tokens 0..2 Query 3 [ 0.10, 0.25, 0.30, 0.35 ] <-- Token 3 sees Tokens 0..3

In Simple English: Multi-Head Attention & Stopping LLMs From Cheating (Causal Mask)

  • Why Multiple Heads (MHA)? In the sentence "The tired dog refused to cross the busy street because it was scared", Head 1 learns grammar ("tired" describes "dog"), Head 2 solves pronoun links ("it" refers to "dog"), and Head 3 watches location ("street"). Having 8–32 parallel heads lets the model wear 8 different pairs of glasses at once!
  • Causal Masking (Blocking Future Peeking): While training an LLM to predict the next word in "The cat sat on the mat", token #3 ("sat") is forbidden from looking ahead at words #4–6 ("on the mat"). Setting future cells to −∞-\infty turns their Softmax probability to strictly 0.0 — like covering the right side of a page with a sheet of paper while studying!