Day 3:LoRA (Low-Rank Adaptation) Matrix Math & Custom PyTorch Layer From Scratch
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 3: LoRA (Low-Rank Adaptation) Matrix Math & Custom PyTorch Layer From Scratch
1. First-Principles Mathematical Theory
When fine-tuning large language models, full fine-tuning updates all weights: where and . For an attention projection matrix in a 70B model where , each weight matrix contains $67,108,864W_q, W_k, W_v, W_o, W_{\text{gate}}, W_{\text{up}}, W_{\text{down}}$), full fine-tuning requires tracking hundreds of gigabytes of optimizer states.
The Intrinsic Rank Hypothesis
Aghajanyan et al. (2020) and Hu et al. (2021) demonstrated that the weight delta needed to adapt a foundation model to a specific task resides in a subspace with an intrinsic rank that is orders of magnitude smaller than :
Therefore, instead of updating the dense matrix , we factorize it into two low-rank matrices: where:
- is frozen ().
- is a trainable projection down-matrix.
- is a trainable projection up-matrix.
- is the rank hyperparameter ().
- is a scaling hyperparameter, ensuring that changes to do not force re-tuning of the learning rate.
2. Gradient Derivations & Initialization Proof
The Zero-Perturbation Initialization Proof
Theorem: At training step , the LoRA adapter must not alter the outputs of the foundation model.
- Matrix is initialized from a normal distribution (e.g. Kaiming/He initialization).
- Matrix is initialized strictly to zero ().
At : Thus, the foundation model's prior representations are preserved with zero initialization noise.
Backpropagation & Gradient Equations
Let the scalar loss be , and the incoming gradient from the subsequent layer be: The forward pass is: By the multivariable chain rule: Because is detached from the autograd graph, . We update only parameters instead of .
3. Parameter Reduction & Zero-Latency Merging
Parameter Savings Ratio
For and :
Zero-Latency Production Deployment
In production, running two parallel branches ( and ) adds latency and VRAM overhead. Before deploying to production (e.g., in vLLM or Ollama), we merge the adapter weights permanently: We then discard and , deploying as a standard linear layer with zero inference latency penalty.
In Simple English: How LoRA Shrinks 16 Million Weights Down to 131,000 (0.78%)
Look at that rank reduction ratio: :
- Imagine editing a massive 4,096 4,096 city map (16.7 million street pixels). Updating every pixel costs gigabytes of optimizer VRAM!
- LoRA () freezes the original map completely untouched on disk, and lays a tiny transparent tracing sheet made of just two thin strips: (
16 x 4096) and (4096 x 16)! - Total new weights = **$2 \times 16 \times 4096 = 131,072B = 0\Delta W = 0$, so your fine-tune starts with the exact clean intelligence of the base model without random chaos!