Week 7 • Day 1Week 7

Day 1:Week 7, Day 1: Scaled Dot-Product Attention — Q, K, V & The 1/sqrt(d_k) Variance Proof

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 7, Day 1: Scaled Dot-Product Attention — Q, K, V & The 1/sqrt(d_k) Variance Proof


1. Intuition: Attention as a Differentiable Database Lookup

In an RNN, token tt can only access token $1throughacompressedhiddenvectorthrough a compressed hidden vectorh_{t-1}.In∗∗Self−Attention∗∗(∗Vaswanietal.,"AttentionIsAllYouNeed",2017∗),everytokendirectlyquerieseveryothertokeninthesequencein∗∗. In **Self-Attention** (*Vaswani et al., "Attention Is All You Need", 2017*), every token directly queries every other token in the sequence in **O(1)$ path length**.

Think of Attention as a soft, differentiable key-value database:

  • Query (qi∈Rdkq_i \in \mathbb{R}^{d_k}): "What information am I looking for?"
  • Key (kj∈Rdkk_j \in \mathbb{R}^{d_k}): "What information do I contain?"
  • Value (vj∈Rdvv_j \in \mathbb{R}^{d_v}): "Here is my actual content payload if we match."

Given input embeddings X∈RT×dmodelX \in \mathbb{R}^{T \times d_{\text{model}}}, we project XX using three learned weight matrices:

Q=XWQ,K=XWK,V=XWVQ = X W_Q, \quad K = X W_K, \quad V = X W_V

And compute Scaled Dot-Product Attention:

Attention(Q,K,V)=softmax(QKTdk)V\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V

code
Q (T x d_k) x K^T (d_k x T) │ ▼ Raw Scores (T x T) ──► Divide by sqrt(d_k) ──► Softmax (rows sum to 1.0) │ ▼ Attention Weights A (T x T) x V (T x d_v) │ ▼ Output (T x d_v)

2. First-Principles Mathematical Proof: Why Divide by dk\sqrt{d_k}?

Why not just compute softmax(QKT)V\text{softmax}(Q K^T) V? Suppose the components of a Query vector q∈Rdk\mathbf{q} \in \mathbb{R}^{d_k} and Key vector k∈Rdk\mathbf{k} \in \mathbb{R}^{d_k} are independent random variables with mean $0 and variance \1$: E[qm]=0,Var(qm)=1,E[km]=0,Var(km)=1\mathbb{E}[q_m] = 0, \quad \text{Var}(q_m) = 1, \quad \mathbb{E}[k_m] = 0, \quad \text{Var}(k_m) = 1

Then each product qmkmq_m k_m has mean $0 and variance \1.Thedotproduct. The dot product \mathbf{q} \cdot \mathbf{k} = \sum_{m=1}^{d_k} q_m k_misthesumofis the sum ofd_k$ independent terms, so by linearity of variance:

E[q⋅k]=0,Var(q⋅k)=∑m=1dk1=dk  ⟹  StdDev(q⋅k)=dk\mathbb{E}[\mathbf{q} \cdot \mathbf{k}] = 0, \qquad \text{Var}(\mathbf{q} \cdot \mathbf{k}) = \sum_{m=1}^{d_k} 1 = d_k \implies \text{StdDev}(\mathbf{q} \cdot \mathbf{k}) = \sqrt{d_k}

What Happens Without 1dk\frac{1}{\sqrt{d_k}}?

If dk=128d_k = 128, the standard deviation of dot products is 128≈11.31\sqrt{128} \approx 11.31, meaning raw logits frequently reach +25+25 or −25-25. When passed into softmax(zi)=ezi∑jezj\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}:

  • The largest logit gets probability pmax⁡≈1.0p_{\max} \approx 1.0, and all other tokens get pj≈0.0p_j \approx 0.0 (One-Hot Saturation).
  • The Jacobian of softmax is ∂pi∂zj=pi(δij−pj)\frac{\partial p_i}{\partial z_j} = p_i(\delta_{ij} - p_j). When pi∈{0,1}p_i \in \{0, 1\}, pi(1−pi)=0  ⟹  p_i(1 - p_i) = 0 \implies Vanishing Gradients inside Softmax!

By dividing by dk\sqrt{d_k}: Var(q⋅kdk)=1(dk)2Var(q⋅k)=dkdk=1.0\text{Var}\left(\frac{\mathbf{q} \cdot \mathbf{k}}{\sqrt{d_k}}\right) = \frac{1}{(\sqrt{d_k})^2} \text{Var}(\mathbf{q} \cdot \mathbf{k}) = \frac{d_k}{d_k} = 1.0

The variance of attention logits stays strictly $1.0regardlessofheaddimensionregardless of head dimensiond_k$!


In Simple English: Attention = Library Book Search + Why Divide by dk\sqrt{d_k}?

Don't let the Greek letters scare you! Imagine walking into a giant library looking for a book about baking bread:

  • Query (QQ): Your sticky note — "I want recipes for bread."
  • Key (KK): The title/tags written on the spine of every book — "Python guide", "Sourdough baking", "Car repair".
  • Value (VV): The actual pages inside each book!
  1. Compare your Query against every book's Key (dot product Q⋅KTQ \cdot K^T). High match for "Sourdough baking"!
  2. Pass matches through Softmax so they become clean weights adding up to 100% (92% Sourdough book + 8% Pastry book + 0% Car repair).
  3. Blend the Values (VV) by those weights to get your answer!

Why Divide by dk\sqrt{d_k} With Tiny Numbers?

Suppose each vector has dk=64d_k = 64 numbers instead of $2.Whenyoumultiplyandadd64numberstogether,therawsumswingswildly(say‘+28‘vs‘+2‘)!Passing‘+28‘intoSoftmaxgives∗∗99.99999. When you multiply and add 64 numbers together, the raw sum swings wildly (say `+28` vs `+2`)! Passing `+28` into Softmax gives **99.99999% to one word and 0% to everything else** — freezing learning (**Vanishing Gradient**). Dividing by \sqrt{64} = 8$ brings +28 gently back to +3.5, keeping Softmax smooth and trainable!