Week 21 • Day 2Week 21

Day 2:Curating High-Quality Supervised Fine-Tuning (SFT) Datasets, ChatML & Loss Masking

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 2: Curating High-Quality Supervised Fine-Tuning (SFT) Datasets, ChatML & Loss Masking

1. The Senior Data Engineering Philosophy: Quality > Quantity

In the early days of LLM fine-tuning, teams dumped 100,000 low-quality scraped web dialogues into training scripts. The result was often model degradation, stylistic incoherence, and hallucination loops.

The seminal LIMA (Less Is More for Alignment) paper (Zhou et al., 2023) revolutionized fine-tuning:

A foundation model already possesses vast world knowledge from pre-training. Supervised Fine-Tuning is simply the process of teaching the model the format, style, and persona in which to surface that knowledge. 1,000 meticulously curated, high-diversity, human-verified instruction pairs routinely outperform 50,000 noisy crowd-sourced samples.


2. Tokenization & ChatML Standard

To fine-tune modern open models (LLaMA-3, Qwen-2.5, Mistral), conversations must be tokenized into standardized multi-turn dialogue templates. The industry standard format is ChatML:

text
<|im_start|>system You are an expert Site Reliability Engineering (SRE) triage copilot.<|im_end|> <|im_start|>user Database CPU utilization reached 98% and pool exhausted. Suggest mitigation.<|im_end|> <|im_start|>assistant Execute connection pool scale-out and kill idle queries: 1. `pg_terminate_backend(pid)` for idle connections > 300s. 2. Increase max_connections to 500.<|im_end|>

Special tokens (<|im_start|>, <|im_end|>) act as control tokens. Foundation models learn to treat anything following <|im_start|>assistant as the generation target.


3. Mathematical Formulation of Assistant-Only Loss Masking

In standard causal language modeling, the loss is the cross-entropy over the sequence of tokens: Lfull(θ)=−1T∑t=1Tlog⁡Pθ(xt∣x<t)\mathcal{L}_{\text{full}}(\theta) = -\frac{1}{T} \sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t})

The Catastrophic Flaw of Full-Sequence Loss

If we compute loss over the entire conversation, the model receives gradients penalizing it for not predicting the user prompt or system prompt!

  • The model wastes capacity memorizing how the user phrased the question.
  • The model develops superstitious associations between specific prompt wording and output accuracy.

The Solution: Target Label Masking (ignore_index = -100)

PyTorch's torch.nn.CrossEntropyLoss includes a parameter ignore_index=-100. Any label token set to -100 is completely omitted from the loss computation and backward pass:

LSFT(θ)=−∑t=1Tmtlog⁡Pθ(xt∣x<t)∑t=1Tmt\mathcal{L}_{\text{SFT}}(\theta) = -\frac{\sum_{t=1}^{T} m_t \log P_\theta(x_t \mid x_{<t})}{\sum_{t=1}^{T} m_t} where the binary mask mtm_t is defined as: mt={1if token xt belongs to the assistant’s turn0if token xt belongs to system prompt, user prompt, or control tagsm_t = \begin{cases} 1 & \text{if token } x_t \text{ belongs to the assistant's turn} \\ 0 & \text{if token } x_t \text{ belongs to system prompt, user prompt, or control tags} \end{cases}

Architecture Flow Diagram
<|im_start|>system ... <|im_end|>
<|im_start|>user ... <|im_end|>
<|im_start|>assistant ... <|im_end|>
[-100, -100, -100, ...] (Masked Out)
[-100, -100, -100, ...] (Masked Out)
[token_id_1, token_id_2, ...] (Active Loss)
LS
LU
LA
LOSS[CrossEntropyLoss(ignore_index=-100)

4. Dataset Hygiene: Deduplication & Decontamination

1. MinHash Deduplication

Redundant examples skew the optimization landscape, causing the model to overfit on repetitive phrases. We compute MinHash signatures and calculate Jaccard similarity: J(D1,D2)=∣D1∩D2∣∣D1∪D2∣J(D_1, D_2) = \frac{|D_1 \cap D_2|}{|D_1 \cup D_2|} Any example pair with J(D1,D2)≥0.85J(D_1, D_2) \ge 0.85 is pruned.

2. Decontamination (Test Set Leakage Defense)

A frequent flaw in enterprise fine-tuning is test set contamination: training examples containing 13-gram overlaps with the final evaluation benchmark. A strict decontamination filter scans and excludes contaminated candidates before training.


In Simple English: Why Assistant-Only Loss Masking (mt=0m_t = 0 on User Prompts) Matters

Look at the mask variable mt∈{0,1}m_t \in \{0, 1\} in LSFT\mathcal{L}_{\text{SFT}}:

  • During fine-tuning, a training sample contains [System Prompt + User Question + Assistant Answer].
  • Setting mt=0m_t = 0 on System & User tokens tells the trainer: "Do NOT grade the model on memorizing the user's question! Compute loss and update weights ONLY (mt=1m_t = 1) on the Assistant's ideal answer tokens!" That focuses 100% of training signal on teaching how to reply, not wasting capacity predicting questions!