Day 6:Preference Alignment — RLHF vs. Direct Preference Optimization (DPO)
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 6: Preference Alignment — RLHF vs. Direct Preference Optimization (DPO)
1. Why Supervised Fine-Tuning (SFT) Is Not Enough
Supervised Fine-Tuning (SFT) trains a model via maximum likelihood estimation (cross-entropy): While SFT teaches the model how to speak (domain vocabulary, ChatML tokens, structured JSON schemas), it suffers from fundamental limitations:
- Mode Collapse & Hallucination Encouragement: The cross-entropy objective penalizes low probabilities on ground truth tokens, forcing the model to assign mass to everything in the training set, even ambiguous or contradictory demonstrations.
- Inability to Discriminate Degrees of Quality: If prompt has two possible answers, one mediocre and one exceptional, SFT simply averages their distributions. It has no mechanism to prefer the better response.
Preference Alignment trains the model on comparative feedback: given prompt , human or expert evaluators indicate that response (winning/chosen) is preferred over (losing/rejected).
2. The RLHF Architecture & Its Engineering Fragility
The classic Reinforcement Learning from Human Feedback (RLHF) pipeline (Christiano et al., 2017; Ouyang et al., 2022) uses Proximal Policy Optimization (PPO):
Why Production Teams Hate PPO
- Extreme VRAM Consumption: Housing 4 distinct models in GPU memory requires multi-node GPU clusters.
- Sampling Bottleneck: PPO is an on-policy algorithm requiring online generation of completions during training, creating massive I/O and GPU idle latency.
- Reward Hacking & Instability: Policies rapidly discover adversarial tokens that trigger high reward model scores while producing gibberish.
3. Direct Preference Optimization (DPO) Derivation
In 2023, Rafael Rafailov et al. (Stanford University) published Direct Preference Optimization: Your Language Model is Secretly a Reward Model, proving that no separate reward model or RL loop is necessary.
Step 1: The RLHF Optimization Objective
Under the standard KL-constrained RL objective: The closed-form analytical solution for the optimal policy is: where is the partition function.
Step 2: Inverting Policy to Express the Implicit Reward
Taking the natural logarithm of both sides and rearranging for the ground-truth reward :
Step 3: Bradley-Terry Preference Modeling
The Bradley-Terry model defines the probability that response is preferred over : Substituting our inverted reward function: The intractable partition function cancels out entirely!
Step 4: The Closed-Form DPO Loss Function
By minimizing the negative log-likelihood of preference pairs, we arrive at the DPO loss:
4. Key DPO Hyperparameters
- ** (Beta, typically $0.1 to \0.5\beta\pi_{\text{ref}}\beta$ allows aggressive pursuit of preferred responses.
- Implicit Reward Margin: During training, the implicit reward margin should increase steadily from to positive values (), indicating successful alignment.
In Simple English: Teaching Good Taste (RLHF vs DPO Winner-vs-Loser Pairs)
- SFT teaches "how to talk like a helpful assistant". Preference Alignment teaches judgment: refusing dangerous hacks, staying concise, and sounding polished!
- Old Way (
RLHF + PPO): Requires juggling 4 giant models in GPU VRAM at once (Actor + Critic + Reference + Reward Model) — notoriously unstable and expensive. - Modern Way (
DPO): Takes pairs of(Prompt, Winner Response y_w, Loser Response y_l)and directly increases the probability gap favoring over while keeping close enough to the reference model so grammar never collapses!