Week 21 • Day 6Week 21

Day 6:Preference Alignment — RLHF vs. Direct Preference Optimization (DPO)

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 6: Preference Alignment — RLHF vs. Direct Preference Optimization (DPO)

1. Why Supervised Fine-Tuning (SFT) Is Not Enough

Supervised Fine-Tuning (SFT) trains a model via maximum likelihood estimation (cross-entropy): max⁡θ∑(x,y)log⁡πθ(y∣x)\max_\theta \sum_{(x, y)} \log \pi_\theta(y \mid x) While SFT teaches the model how to speak (domain vocabulary, ChatML tokens, structured JSON schemas), it suffers from fundamental limitations:

  1. Mode Collapse & Hallucination Encouragement: The cross-entropy objective penalizes low probabilities on ground truth tokens, forcing the model to assign mass to everything in the training set, even ambiguous or contradictory demonstrations.
  2. Inability to Discriminate Degrees of Quality: If prompt xx has two possible answers, one mediocre and one exceptional, SFT simply averages their distributions. It has no mechanism to prefer the better response.

Preference Alignment trains the model on comparative feedback: given prompt xx, human or expert evaluators indicate that response ywy_w (winning/chosen) is preferred over yly_l (losing/rejected).


2. The RLHF Architecture & Its Engineering Fragility

The classic Reinforcement Learning from Human Feedback (RLHF) pipeline (Christiano et al., 2017; Ouyang et al., 2022) uses Proximal Policy Optimization (PPO):

Architecture Flow Diagram
Actor Policy π_θ (Generates tokens, updated by PPO)
Critic / Value Network V_ϕ (Estimates state value)
Reference Policy π_ref (Frozen, prevents mode collapse)
Reward Model R_ψ (Frozen scalar scoring network)
ROLLOUT[On-Policy Rollout Generation
M4
REWARD[Reward Score r(x, y)
M3
KL[KL Penalty β D_KL(π_θ || π_ref)
PPO_OPT[PPO Generalized Advantage Estimation (GAE)
PPO_OPT
PPO_OPT
UPDATE[Gradient Update to π_θ and V_ϕ

Why Production Teams Hate PPO

  • Extreme VRAM Consumption: Housing 4 distinct models in GPU memory requires multi-node GPU clusters.
  • Sampling Bottleneck: PPO is an on-policy algorithm requiring online generation of completions during training, creating massive I/O and GPU idle latency.
  • Reward Hacking & Instability: Policies rapidly discover adversarial tokens that trigger high reward model scores while producing gibberish.

3. Direct Preference Optimization (DPO) Derivation

In 2023, Rafael Rafailov et al. (Stanford University) published Direct Preference Optimization: Your Language Model is Secretly a Reward Model, proving that no separate reward model or RL loop is necessary.

Step 1: The RLHF Optimization Objective

Under the standard KL-constrained RL objective: max⁡πEx∼D,y∼π[r(x,y)]−βDKL(π(y∣x)∥πref(y∣x))\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi} \left[ r(x, y) \right] - \beta D_{\text{KL}}(\pi(y \mid x) \parallel \pi_{\text{ref}}(y \mid x)) The closed-form analytical solution for the optimal policy π∗(y∣x)\pi^*(y \mid x) is: π∗(y∣x)=1Z(x)πref(y∣x)exp⁡(1βr(x,y))\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\left( \frac{1}{\beta} r(x, y) \right) where Z(x)=∑yπref(y∣x)exp⁡(1βr(x,y))Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp\left( \frac{1}{\beta} r(x, y) \right) is the partition function.

Step 2: Inverting Policy to Express the Implicit Reward

Taking the natural logarithm of both sides and rearranging for the ground-truth reward r(x,y)r(x, y): r(x,y)=βlog⁡π∗(y∣x)πref(y∣x)+βlog⁡Z(x)r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)

Step 3: Bradley-Terry Preference Modeling

The Bradley-Terry model defines the probability that response ywy_w is preferred over yly_l: P(yw≻yl∣x)=σ(r(x,yw)−r(x,yl))P(y_w \succ y_l \mid x) = \sigma\left( r(x, y_w) - r(x, y_l) \right) Substituting our inverted reward function: r(x,yw)−r(x,yl)=(βlog⁡π(yw∣x)πref(yw∣x)+βlog⁡Z(x))−(βlog⁡π(yl∣x)πref(yl∣x)+βlog⁡Z(x))r(x, y_w) - r(x, y_l) = \left( \beta \log \frac{\pi(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} + \beta \log Z(x) \right) - \left( \beta \log \frac{\pi(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} + \beta \log Z(x) \right) The intractable partition function Z(x)Z(x) cancels out entirely! r(x,yw)−r(x,yl)=β(log⁡π(yw∣x)πref(yw∣x)−log⁡π(yl∣x)πref(yl∣x))r(x, y_w) - r(x, y_l) = \beta \left( \log \frac{\pi(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \log \frac{\pi(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right)

Step 4: The Closed-Form DPO Loss Function

By minimizing the negative log-likelihood of preference pairs, we arrive at the DPO loss: LDPO(θ;πref)=−E(x,yw,yl)∼D[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\text{DPO}}(\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]

Architecture Flow Diagram
Prompt x->P_THETA[Current Policy π_θ
P_REF[Reference Policy π_ref (Frozen)
LOGP_W[log π_θ(y_w | x)
LOGP_L[log π_θ(y_l | x)
REFP_W[log π_ref(y_w | x)
REFP_L[log π_ref(y_l | x)
SUB1((-)
SUB1
R_W[Implicit Reward r̂_θ(x, y_w)
SUB2((-)
SUB2
R_L[Implicit Reward r̂_θ(x, y_l)
SUB_MARGIN((-)
SUB_MARGIN

4. Key DPO Hyperparameters

  1. **β\beta (Beta, typically $0.1 to \0.5)∗∗:ControlsthestrengthoftheKLconstraintagainstthereferencemodel.Ahigher)**: Controls the strength of the KL constraint against the reference model. A higher \betapreventsthemodelfromdeviatingfromprevents the model from deviating from\pi_{\text{ref}},whilealower, while a lower \beta$ allows aggressive pursuit of preferred responses.
  2. Implicit Reward Margin: Margin=r^θ(x,yw)−r^θ(x,yl)=β(log⁡πθ(yw∣x)πref(yw∣x)−log⁡πθ(yl∣x)πref(yl∣x))\text{Margin} = \hat{r}_\theta(x, y_w) - \hat{r}_\theta(x, y_l) = \beta \left( \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) During training, the implicit reward margin should increase steadily from ≈0.0\approx 0.0 to positive values (>1.0> 1.0), indicating successful alignment.

In Simple English: Teaching Good Taste (RLHF vs DPO Winner-vs-Loser Pairs)

  • SFT teaches "how to talk like a helpful assistant". Preference Alignment teaches judgment: refusing dangerous hacks, staying concise, and sounding polished!
  • Old Way (RLHF + PPO): Requires juggling 4 giant models in GPU VRAM at once (Actor + Critic + Reference + Reward Model) — notoriously unstable and expensive.
  • Modern Way (DPO): Takes pairs of (Prompt, Winner Response y_w, Loser Response y_l) and directly increases the probability gap favoring ywy_w over yly_l while keeping β\beta close enough to the reference model so grammar never collapses!