Week 21 • Day 3Week 21

Day 3:LoRA (Low-Rank Adaptation) Matrix Math & Custom PyTorch Layer From Scratch

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 3: LoRA (Low-Rank Adaptation) Matrix Math & Custom PyTorch Layer From Scratch

1. First-Principles Mathematical Theory

When fine-tuning large language models, full fine-tuning updates all weights: W=W0+ΔWW = W_0 + \Delta W where W0∈Rd×kW_0 \in \mathbb{R}^{d \times k} and ΔW∈Rd×k\Delta W \in \mathbb{R}^{d \times k}. For an attention projection matrix in a 70B model where d=k=8192d = k = 8192, each weight matrix contains $67,108,864parameters(over134MBinFP16).Across80transformerlayersandallprojectionmatrices( parameters (over 134 MB in FP16). Across 80 transformer layers and all projection matrices (W_q, W_k, W_v, W_o, W_{\text{gate}}, W_{\text{up}}, W_{\text{down}}$), full fine-tuning requires tracking hundreds of gigabytes of optimizer states.

The Intrinsic Rank Hypothesis

Aghajanyan et al. (2020) and Hu et al. (2021) demonstrated that the weight delta ΔW\Delta W needed to adapt a foundation model to a specific task resides in a subspace with an intrinsic rank rr that is orders of magnitude smaller than min⁡(d,k)\min(d, k): rank⁡(ΔW)≤r≪min⁡(d,k)\operatorname{rank}(\Delta W) \le r \ll \min(d, k)

Therefore, instead of updating the dense matrix ΔW\Delta W, we factorize it into two low-rank matrices: ΔW=αrBA\Delta W = \frac{\alpha}{r} B A where:

  • W0∈Rd×kW_0 \in \mathbb{R}^{d \times k} is frozen (∇W0L=0\nabla_{W_0} \mathcal{L} = 0).
  • A∈Rr×kA \in \mathbb{R}^{r \times k} is a trainable projection down-matrix.
  • B∈Rd×rB \in \mathbb{R}^{d \times r} is a trainable projection up-matrix.
  • rr is the rank hyperparameter (r∈{4,8,16,32}r \in \{4, 8, 16, 32\}).
  • α\alpha is a scaling hyperparameter, ensuring that changes to rr do not force re-tuning of the learning rate.
Architecture Flow Diagram
Input x ∈ ℝ^k->W0[W₀ ∈ ℝ^(d × k) [FROZEN]
W0X[W₀ x ∈ ℝ^d
A[Matrix A ∈ ℝ^(r × k)<br/>[Gaussian Init: 𝒩(0, σ²)]
AX[A x ∈ ℝ^r (Down-Projected)
B[Matrix B ∈ ℝ^(d × r)<br/>[Zero Init: 0]
BAX[B A x ∈ ℝ^d (Up-Projected)
SCALE[Scale by (α / r)
ADD((+)
ADD
H[Output Activation h = W₀ x + (α / r) B A x

2. Gradient Derivations & Initialization Proof

The Zero-Perturbation Initialization Proof

Theorem: At training step t=0t = 0, the LoRA adapter must not alter the outputs of the foundation model.

  • Matrix AA is initialized from a normal distribution N(0,σ2)\mathcal{N}(0, \sigma^2) (e.g. Kaiming/He initialization).
  • Matrix BB is initialized strictly to zero (B=0B = \mathbf{0}).

At t=0t = 0: ΔW0=αrBA=αr(0⋅A)=0\Delta W_0 = \frac{\alpha}{r} B A = \frac{\alpha}{r} (\mathbf{0} \cdot A) = \mathbf{0} h=W0x+ΔW0x=W0x+0=W0xh = W_0 x + \Delta W_0 x = W_0 x + \mathbf{0} = W_0 x Thus, the foundation model's prior representations are preserved with zero initialization noise.

Backpropagation & Gradient Equations

Let the scalar loss be L\mathcal{L}, and the incoming gradient from the subsequent layer be: δ=∂L∂h∈Rd\delta = \frac{\partial \mathcal{L}}{\partial h} \in \mathbb{R}^{d} The forward pass is: h=W0x+αrBAxh = W_0 x + \frac{\alpha}{r} B A x By the multivariable chain rule: ∂L∂B=αrδ(Ax)T∈Rd×r\frac{\partial \mathcal{L}}{\partial B} = \frac{\alpha}{r} \delta (A x)^T \in \mathbb{R}^{d \times r} ∂L∂A=αr(BTδ)xT∈Rr×k\frac{\partial \mathcal{L}}{\partial A} = \frac{\alpha}{r} (B^T \delta) x^T \in \mathbb{R}^{r \times k} Because W0W_0 is detached from the autograd graph, ∂L∂W0=0\frac{\partial \mathcal{L}}{\partial W_0} = \mathbf{0}. We update only r(d+k)r(d + k) parameters instead of d⋅kd \cdot k.


3. Parameter Reduction & Zero-Latency Merging

Parameter Savings Ratio

Reduction Ratio=r(d+k)d⋅k=r(1d+1k)\text{Reduction Ratio} = \frac{r(d + k)}{d \cdot k} = r \left( \frac{1}{d} + \frac{1}{k} \right) For d=k=4096d = k = 4096 and r=16r = 16: Reduction=16×(4096+4096)4096×4096=131,07216,777,216≈0.00781(99.22% reduction!)\text{Reduction} = \frac{16 \times (4096 + 4096)}{4096 \times 4096} = \frac{131,072}{16,777,216} \approx 0.00781 \quad (\mathbf{99.22\% \text{ reduction!}})

Zero-Latency Production Deployment

In production, running two parallel branches (W0xW_0 x and αrBAx\frac{\alpha}{r} B A x) adds latency and VRAM overhead. Before deploying to production (e.g., in vLLM or Ollama), we merge the adapter weights permanently: Wmerged=W0+αrBAW_{\text{merged}} = W_0 + \frac{\alpha}{r} B A We then discard AA and BB, deploying WmergedW_{\text{merged}} as a standard linear layer with zero inference latency penalty.


In Simple English: How LoRA Shrinks 16 Million Weights Down to 131,000 (0.78%)

Look at that rank reduction ratio: 16×(4096+4096)4096×4096=0.78%\frac{16 \times (4096 + 4096)}{4096 \times 4096} = \mathbf{0.78\%}:

  • Imagine editing a massive 4,096 ×\times 4,096 city map (16.7 million street pixels). Updating every pixel costs gigabytes of optimizer VRAM!
  • LoRA (W0+αrBAW_0 + \frac{\alpha}{r} B A) freezes the original map W0W_0 completely untouched on disk, and lays a tiny transparent tracing sheet made of just two thin strips: AA (16 x 4096) and BB (4096 x 16)!
  • Total new weights = **$2 \times 16 \times 4096 = 131,072∗∗—∗∗over128xfewertrainableweights∗∗!Andinitializing** — **over 128x fewer trainable weights**! And initializing B = 0meansatStep0,means at Step 0,\Delta W = 0$, so your fine-tune starts with the exact clean intelligence of the base model without random chaos!