Day 1:Week 7, Day 1: Scaled Dot-Product Attention — Q, K, V & The 1/sqrt(d_k) Variance Proof
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 7, Day 1: Scaled Dot-Product Attention — Q, K, V & The 1/sqrt(d_k) Variance Proof
1. Intuition: Attention as a Differentiable Database Lookup
In an RNN, token can only access token $1h_{t-1}O(1)$ path length**.
Think of Attention as a soft, differentiable key-value database:
- Query (): "What information am I looking for?"
- Key (): "What information do I contain?"
- Value (): "Here is my actual content payload if we match."
Given input embeddings , we project using three learned weight matrices:
And compute Scaled Dot-Product Attention:
Q (T x d_k) x K^T (d_k x T)
│
▼
Raw Scores (T x T) ──► Divide by sqrt(d_k) ──► Softmax (rows sum to 1.0)
│
▼
Attention Weights A (T x T) x V (T x d_v)
│
▼
Output (T x d_v)
2. First-Principles Mathematical Proof: Why Divide by ?
Why not just compute ? Suppose the components of a Query vector and Key vector are independent random variables with mean $0 and variance \1$:
Then each product has mean $0 and variance \1\mathbf{q} \cdot \mathbf{k} = \sum_{m=1}^{d_k} q_m k_md_k$ independent terms, so by linearity of variance:
What Happens Without ?
If , the standard deviation of dot products is , meaning raw logits frequently reach or . When passed into :
- The largest logit gets probability , and all other tokens get (One-Hot Saturation).
- The Jacobian of softmax is . When , Vanishing Gradients inside Softmax!
By dividing by :
The variance of attention logits stays strictly $1.0d_k$!
In Simple English: Attention = Library Book Search + Why Divide by ?
Don't let the Greek letters scare you! Imagine walking into a giant library looking for a book about baking bread:
- Query (): Your sticky note — "I want recipes for bread."
- Key (): The title/tags written on the spine of every book — "Python guide", "Sourdough baking", "Car repair".
- Value (): The actual pages inside each book!
- Compare your Query against every book's Key (dot product ). High match for "Sourdough baking"!
- Pass matches through Softmax so they become clean weights adding up to 100% (
92% Sourdough book + 8% Pastry book + 0% Car repair). - Blend the Values () by those weights to get your answer!
Why Divide by With Tiny Numbers?
Suppose each vector has numbers instead of $2\sqrt{64} = 8$ brings +28 gently back to +3.5, keeping Softmax smooth and trainable!