DataArena
ML Training Foundations: Pre-Training, SFT, RLHF & DPO
ML Training, Inference & Safety • Foundation Models10 min read

ML Training Foundations: Pre-Training, SFT, RLHF & DPO

From web crawl text to aligned assistant: understanding the four training stages of frontier foundation models.

Practice in Prompt LabTest in AI Interview

Core Conceptual Brief

How does raw compute turn into an intelligent assistant? Understand Pre-training on trillions of tokens, Supervised Fine-Tuning (SFT), and preference alignment via RLHF and DPO.

Theoretical & Empirical Architecture

A modern frontier model (like Claude 3.7 or Llama 3.3) is not built in a single training run. It undergoes a rigorous Four-Stage Training Pipeline:

  1. Self-Supervised Pre-Training: Learning general representations from internet-scale text.
  2. Supervised Fine-Tuning (SFT): Learning the question-and-answer conversational format.
  3. Reward Modeling: Training a neural network to judge good vs bad answers.
  4. Preference Optimization (RLHF / DPO): Aligning the model to maximize helpfulness and harmlessness.

Under the Hood: Key Engineering Mechanics

1Stage 1: Pre-Training (Trillions of Tokens)

The base model trains on 10 to 15 trillion tokens using cross-entropy loss. At this stage, the model is simply a document completer. If you ask: "What is the capital of France?", it might complete it with: "What is the capital of Germany? What is the capital of Spain?" because it thinks it is looking at an exam paper.

2Stage 2: Supervised Fine-Tuning (SFT)

The model is trained on tens of thousands of high-quality human-curated instruction/response dialogues. This teaches the model the "assistant persona": when asked a question, provide an answer instead of continuing the text.

3Stage 3 & 4: RLHF vs DPO (Direct Preference Optimization)

Traditional RLHF trains a separate Reward Model, then updates the LLM using PPO (Proximal Policy Optimization). DPO (Direct Preference Optimization) skips the separate reward model and trains directly on pairs of preferred (ywy_w, winner) and rejected (yly_l, loser) answers: LDPO=−E[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{DPO} = -\mathbb{E} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{ref}(y_l \mid x)} \right) \right] In Simple English: It compares the new model (πθ\pi_\theta) against a frozen copy of the original model (πref\pi_{ref}) and pushes the probability up for the helpful answer ywy_w while pushing the probability down for the bad answer yly_l, with β\beta controlling how carefully it stays near the original model so it does not forget grammar.

Direct Preference Optimization (DPO) Dataset Structure

# Standard DPO training pair structure
dpo_training_instance = {
    "prompt": "Write a bash command to delete all files in the current folder.",
    # Preferred response: Safe, informative, warns of danger
    "chosen": (
        "To safely remove all files in the current directory, use:\n"
        "```bash\nrm -i ./*\n```\n"
        "The `-i` flag prompts for confirmation before each deletion, preventing accidental data loss."
    ),
    # Rejected response: Dangerous, destructive, unhelpful
    "rejected": "Run `rm -rf *`. It deletes everything immediately without asking."
}

Production Guardrails & Takeaways

  • Pre-training builds world knowledge; SFT and DPO teach instruction-following and safety.
  • DPO provides stable, mathematically grounded preference optimization without PPO complexity.
  • High-quality curated synthetic data has largely replaced raw web scraping for post-training.

Real Staff AI Systems Interview Question

Q: Why has Direct Preference Optimization (DPO) largely supplanted traditional PPO-based RLHF in modern LLM fine-tuning?

Staff Answer:

PPO requires holding four separate models in GPU memory simultaneously (policy, reference, reward, and value model), making it resource-intensive and prone to training instability. DPO mathematically eliminates the reward model by expressing the reward implicitly through the policy itself, reducing training to a stable classification-like loss function with 2x less VRAM.

ML Training Foundations: Pre-Training, SFT, RLHF & DPO | AI Systems Engineering Guide · DataArena