Week 7, Day 3: Positional Encodings — Sinusoidal vs Rotary Position Embeddings (RoPE)

1. Why Unmodified Self-Attention is a "Bag of Words"

Look closely at $\text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$. If you swap the order of tokens in a sentence ("dog bites man" vs "man bites dog"), the rows of $Q, K, V$ simply permute in the exact same way (Permutation Equivariance). Without positional information, a Transformer cannot tell whether a word appeared at position 1 or position 1,000!

2. Generation 1: Additive Sinusoidal Positional Encodings (2017)

The original 2017 Transformer added a fixed vector $PE_{(pos)} \in \mathbb{R}^{d_{\text{model}}}$ directly to the input token embeddings $X + PE$:

$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i / d_{\text{model}}}}\right), \qquad PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i / d_{\text{model}}}}\right)$

Limitation of Additive Encodings:

• Positional information is only injected at Layer 0 and gets diluted after passing through dozens of linear projections.
• It encodes absolute position $pos$, whereas attention dot products $q_m \cdot k_n$ fundamentally care about the relative distance $(m - n)$ between Query token $m$ and Key token $n$.

3. Generation 2 (Modern Standard): Rotary Position Embeddings (RoPE)

Used in Llama-3, Qwen, Mistral, Gemma, PaLM, DeepSeek (Su et al., "RoFormer", 2021). Instead of adding a vector to embeddings at Layer 0, RoPE rotates Query $q_m$ and Key $k_n$ inside every attention layer by an angle proportional to their position index $m\theta_i$ and $n\theta_i$!

The $2 \times 2$ Rotation Matrix Derivation

Group each head's $d_k$ dimensions into $d_k / 2$ two-dimensional pairs $(x^{(1)}, x^{(2)})$. Rotate the 2D vector at token position $m$ by angle $m\theta$:

$R_{\Theta, m} \mathbf{x} = \begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix} \begin{pmatrix} x^{(1)} \\ x^{(2)} \end{pmatrix}$

Why RoPE is Mathematically Brilliant: Relative Distance Invariance!

When we compute the attention dot product between rotated Query at position $m$ and rotated Key at position $n$:

$\langle R_m \mathbf{q}, R_n \mathbf{k} \rangle = \mathbf{q}^T R_m^T R_n \mathbf{k} = \mathbf{q}^T R_{n - m} \mathbf{k}$

Because the transpose of a rotation by $m\theta$ followed by a rotation by $n\theta$ is simply a rotation by $(n - m)\theta$, the dot product depends ONLY on the relative token distance $(m - n)$! Whether two words are at positions (2, 5) or (1002, 1005) (both distance 3 apart), their RoPE attention dot product is 100% identical!

In Simple English: Why Transformers Need Word Order (RoPE Clock Hands)

Self-Attention looks at all words at the exact same time — so without position numbers, "Dog bites man" and "Man bites dog" would look identical!

• Old Fix (Sinusoidal): Stamp a unique barcode wave onto each slot (#1, #2, #3...). Downside: works poorly if tested on longer texts than training.
• Modern Fix (RoPE — Rotary Position Embeddings): Think of each pair of numbers in a word's vector as a clock hand! Word at position 1 turns the hand by $1\theta; word at position 5 turns it by \5\theta. When you take the dot product between word 1 and word 5, the angle depends ONLY on their **gap (\5 - 1 = 4$ words apart)** — letting Llama, Qwen, and Claude generalize naturally across long contexts!