Week 21 • Day 5Week 21

Day 5:Hands-On PEFT & TRL Training Pipeline and Loss Diagnostics

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 5: Hands-On PEFT & TRL Training Pipeline and Loss Diagnostics

1. The Production Fine-Tuning Stack

In modern production environments, parameter-efficient fine-tuning leverages the Hugging Face ecosystem:

  • transformers: Foundation model architectures (LLaMA, Qwen, Mistral) and tokenizers.
  • peft (Parameter-Efficient Fine-Tuning): Injects LoRA, AdaLoRA, PrefixTuning, or PromptTuning adapters into target projection matrices (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).
  • trl (Transformer Reinforcement Learning): Provides SFTTrainer and DPOTrainer with native ChatML packing, dataset tokenization, and loss masking.
  • bitsandbytes: 4-bit NormalFloat quantization backend for QLoRA.
Architecture Flow Diagram
Raw Dialogue Turns (System / User / Assistant)
ChatML Formatter (<|im_start|> ... <|im_end|>)
Target Loss Masking (Prompt = -100)
D2->D3
Base Model (e.g., LLaMA-3.1-8B in 4-bit NF4)
LoraConfig(r=16, alpha=32, target_modules=[q_proj, v_proj])
get_peft_model(model, peft_config)
M3
M3
Micro-Batch Forward Pass
CrossEntropyLoss(ignore_index=-100)
Gradient Accumulation (N Steps)
Cosine LR Annealing with Warmup
T2->T3->T4

2. Mathematical Hyperparameter Dynamics

1. Effective Batch Size Formulation

To stabilize SGD/AdamW updates without running out of GPU memory, we decouple the physical GPU batch size from the mathematical optimization batch size via Gradient Accumulation: Batcheffective=Batchper-device×Ngrad_accum×NGPUs\text{Batch}_{\text{effective}} = \text{Batch}_{\text{per-device}} \times N_{\text{grad\_accum}} \times N_{\text{GPUs}} For instruction tuning an 8B model:

  • Batchper-device=2\text{Batch}_{\text{per-device}} = 2 (fits easily in 16GB VRAM)
  • Ngrad_accum=16N_{\text{grad\_accum}} = 16
  • NGPUs=1N_{\text{GPUs}} = 1
  • Batcheffective=2×16×1=32 sequences\text{Batch}_{\text{effective}} = 2 \times 16 \times 1 = \mathbf{32\text{ sequences}}

2. Learning Rate Schedule: Cosine with Linear Warmup

LoRA parameters (AA and BB) are initialized from scratch, requiring a higher learning rate (η≈10−4\eta \approx 10^{-4} to $3 \times 10^{-4})thanfullfine−tuning() than full fine-tuning (\eta \approx 10^{-5}$):

\eta_{\max} \cdot \frac{t}{T_{\text{warmup}}} & \text{if } t < T_{\text{warmup}} \\ \eta_{\min} + \frac{1}{2} (\eta_{\max} - \eta_{\min}) \left( 1 + \cos\left( \pi \frac{t - T_{\text{warmup}}}{T_{\text{total}} - T_{\text{warmup}}} \right) \right) & \text{if } t \ge T_{\text{warmup}} \end{cases}$$ --- ## 3. Loss Curve Diagnostics: How Principal Engineers Debug Training ```mermaid xychart-beta title "SFT Loss Curves: Healthy Convergence vs Pathological Failure Modes" x-axis ["0%", "20%", "40%", "60%", "80%", "100%"] y-axis "Cross Entropy Loss" 0.0 --> 3.0 line [2.8, 1.8, 1.3, 1.0, 0.85, 0.75] line [2.8, 1.6, 1.1, 1.4, 1.9, 2.5] line [2.8, 2.7, 2.65, 2.62, 2.60, 2.58] ``` ### Diagnostic Failure Signatures 1. **Healthy Convergence (Blue)**: Training loss drops steeply in first 20% of steps (learning formatting and tone), then gradually tapers down. Validation loss tracks closely ($|\mathcal{L}_{\text{val}} - \mathcal{L}_{\text{train}}| < 0.2$). 2. **Severe Overfitting (Red)**: Training loss plunges toward zero, while validation loss explodes upward after 40% of training. - *Root Cause*: Rank $r$ too high, training for too many epochs ($> 3$), or dataset lack of diversity. - *Remedy*: Decrease $r$ to 8, add LoRA dropout ($p = 0.1$), or inject general instruction samples. 3. **Severe Underfitting / Plateau (Green)**: Loss barely moves from initial pre-training value. - *Root Cause*: Learning rate too small ($\eta < 10^{-5}$), prompt loss masking is incorrectly masking assistant tokens, or too few target modules (only `q_proj`). - *Remedy*: Increase learning rate to \$2 \times 10^{-4}$, inspect target module list (add `v_proj`, `k_proj`, `o_proj`, `gate_proj`). 4. **Gradient Explosion / Loss Spike**: Loss jumps instantly to `NaN` or $> 10.0$. - *Root Cause*: Learning rate too high, or unclipped outlier gradient norm. - *Remedy*: Enable gradient clipping (`max_grad_norm = 1.0`), switch from FP16 to BF16 (BF16 has identical dynamic range to FP32). --- ## In Simple English: Effective Batch Size & Linear Warmup Look at $\text{Batch}_{\text{effective}} = \text{per\_device} \times \text{grad\_accum} \times \text{num\_gpus}$: - If your single GPU only has enough VRAM for **2 examples at a time**, setting `gradient_accumulation_steps = 8` waits for **8 mini-batches (16 examples total)** before taking 1 smooth weight update step — getting big-cluster training stability on a single affordable GPU! - **Warmup + Cosine Decay:** Don't jump straight to a high learning rate at Step 1 when random LoRA sheets are fresh (you'll wreck pretrained weights!). Ramp up gently (**Warmup**), then glide slowly down to zero (**Cosine Decay**) for crisp convergence!