Week 7, Day 4: Modern Transformer Internals — Residual Streams, Pre-Norm RMSNorm & SwiGLU

1. How Modern LLMs (Llama-3, Qwen, Mistral) Differ from the 2017 Transformer

If you read the 2017 "Attention Is All You Need" paper and compare it to Llama-3, three major architectural upgrades stand out inside every Transformer block:

2. Pre-Norm Residual Stream & RMSNorm Math

Why Pre-Norm Beat Post-Norm

In Pre-Norm, the residual highway $x_{l+1} = x_l + f_l(\text{Norm}(x_l))$ has an unobstructed identity path all the way from the final layer $L$ back to Layer $0$: $\frac{\partial x_L}{\partial x_0} = I + \sum_{l} \frac{\partial f_l}{\partial x_0}$ This completely eliminates the need for fragile learning-rate warmup schedules to avoid early gradient explosion!

RMSNorm (Zhang & Sennrich, 2019)

Research showed that LayerNorm's mean-centering ($x - \mu$) contributes almost nothing to stability while costing extra GPU memory reads/writes. RMSNorm drops mean-centering and normalizes strictly by the Root Mean Square:

$\text{RMS}(\mathbf{x}) = \sqrt{\frac{1}{d} \sum_{i=1}^{d} x_i^2 + \epsilon}, \qquad \text{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\text{RMS}(\mathbf{x})} \odot \boldsymbol{\gamma}$

3. The SwiGLU Feed-Forward Network (Shazeer, 2020)

In every Transformer block, Attention mixes information across tokens, while the Feed-Forward Network (FFN / MLP) processes each token independently across channels (acting as a key-value memory bank storing factual knowledge).

Modern LLMs use SwiGLU (Swish-Gated Linear Unit) with three weight matrices ($W_{\text{gate}}, W_{\text{up}} \in \mathbb{R}^{d \times d_{\text{ff}}}$ and $W_{\text{down}} \in \mathbb{R}^{d_{\text{ff}} \times d}$):

$\text{Swish}_1(z) = \text{SiLU}(z) = z \cdot \sigma(z) = \frac{z}{1 + e^{-z}}$ $\text{SwiGLU}(\mathbf{x}) = \Big(\text{SiLU}(\mathbf{x} W_{\text{gate}}) \odot (\mathbf{x} W_{\text{up}})\Big) W_{\text{down}}$

Why the element-wise gate $\odot$? The multiplicative gate allows the FFN to dynamically filter which features pass through for each token, consistently outperforming standard ReLU or GELU MLPs at the same compute budget! To keep total parameter count equal to a 2-matrix FFN with $d_{\text{ff}} = 4d$ ($2 \times 4d^2 = 8d^2$), SwiGLU sets$d_{\text{ff}} = \frac{8}{3}d \approx 2.67d so \3 \times \frac{8}{3}d^2 = 8d^2$.

In Simple English: Highway Bypass (Residual), RMSNorm & SwiGLU

• Residual Stream ($x + f(x)$ — Highway Express Lane): Imagine passing a note down a 32-person line (32 Transformer layers). Without a shortcut, each layer alters the message until gradients vanish. Adding $x + f(x)$ gives information an unbroken highway lane straight from Layer 1 to Layer 32 — each layer just adds subtle refinements!
• Pre-Norm (RMSNorm): Before each layer touches the highway, keep the numbers well-behaved (~radius 1.0) so activations don't explode into Inf. RMSNorm skips subtracting the mean, making it faster on GPUs than older LayerNorm.
• SwiGLU: A gated switch (gate * signal) inside the feedforward block that smoothly decides which features to amplify — delivering ~1–2% better accuracy per FLOP in Llama 3 / Mistral / DeepSeek!