Week 21 • Day 5Week 21
Day 5:Hands-On PEFT & TRL Training Pipeline and Loss Diagnostics
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 5: Hands-On PEFT & TRL Training Pipeline and Loss Diagnostics
1. The Production Fine-Tuning Stack
In modern production environments, parameter-efficient fine-tuning leverages the Hugging Face ecosystem:
transformers: Foundation model architectures (LLaMA, Qwen, Mistral) and tokenizers.peft(Parameter-Efficient Fine-Tuning): InjectsLoRA,AdaLoRA,PrefixTuning, orPromptTuningadapters into target projection matrices (q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj).trl(Transformer Reinforcement Learning): ProvidesSFTTrainerandDPOTrainerwith native ChatML packing, dataset tokenization, and loss masking.bitsandbytes: 4-bit NormalFloat quantization backend for QLoRA.
Architecture Flow Diagram
Raw Dialogue Turns (System / User / Assistant)
ChatML Formatter (<|im_start|> ... <|im_end|>)
Target Loss Masking (Prompt = -100)
D2->D3
Base Model (e.g., LLaMA-3.1-8B in 4-bit NF4)
LoraConfig(r=16, alpha=32, target_modules=[q_proj, v_proj])
get_peft_model(model, peft_config)
M3
M3
Micro-Batch Forward Pass
CrossEntropyLoss(ignore_index=-100)
Gradient Accumulation (N Steps)
Cosine LR Annealing with Warmup
T2->T3->T4
2. Mathematical Hyperparameter Dynamics
1. Effective Batch Size Formulation
To stabilize SGD/AdamW updates without running out of GPU memory, we decouple the physical GPU batch size from the mathematical optimization batch size via Gradient Accumulation: For instruction tuning an 8B model:
- (fits easily in 16GB VRAM)
2. Learning Rate Schedule: Cosine with Linear Warmup
LoRA parameters ( and ) are initialized from scratch, requiring a higher learning rate ( to $3 \times 10^{-4}\eta \approx 10^{-5}$):
\eta_{\max} \cdot \frac{t}{T_{\text{warmup}}} & \text{if } t < T_{\text{warmup}} \\ \eta_{\min} + \frac{1}{2} (\eta_{\max} - \eta_{\min}) \left( 1 + \cos\left( \pi \frac{t - T_{\text{warmup}}}{T_{\text{total}} - T_{\text{warmup}}} \right) \right) & \text{if } t \ge T_{\text{warmup}} \end{cases}$$ --- ## 3. Loss Curve Diagnostics: How Principal Engineers Debug Training ```mermaid xychart-beta title "SFT Loss Curves: Healthy Convergence vs Pathological Failure Modes" x-axis ["0%", "20%", "40%", "60%", "80%", "100%"] y-axis "Cross Entropy Loss" 0.0 --> 3.0 line [2.8, 1.8, 1.3, 1.0, 0.85, 0.75] line [2.8, 1.6, 1.1, 1.4, 1.9, 2.5] line [2.8, 2.7, 2.65, 2.62, 2.60, 2.58] ``` ### Diagnostic Failure Signatures 1. **Healthy Convergence (Blue)**: Training loss drops steeply in first 20% of steps (learning formatting and tone), then gradually tapers down. Validation loss tracks closely ($|\mathcal{L}_{\text{val}} - \mathcal{L}_{\text{train}}| < 0.2$). 2. **Severe Overfitting (Red)**: Training loss plunges toward zero, while validation loss explodes upward after 40% of training. - *Root Cause*: Rank $r$ too high, training for too many epochs ($> 3$), or dataset lack of diversity. - *Remedy*: Decrease $r$ to 8, add LoRA dropout ($p = 0.1$), or inject general instruction samples. 3. **Severe Underfitting / Plateau (Green)**: Loss barely moves from initial pre-training value. - *Root Cause*: Learning rate too small ($\eta < 10^{-5}$), prompt loss masking is incorrectly masking assistant tokens, or too few target modules (only `q_proj`). - *Remedy*: Increase learning rate to \$2 \times 10^{-4}$, inspect target module list (add `v_proj`, `k_proj`, `o_proj`, `gate_proj`). 4. **Gradient Explosion / Loss Spike**: Loss jumps instantly to `NaN` or $> 10.0$. - *Root Cause*: Learning rate too high, or unclipped outlier gradient norm. - *Remedy*: Enable gradient clipping (`max_grad_norm = 1.0`), switch from FP16 to BF16 (BF16 has identical dynamic range to FP32). --- ## In Simple English: Effective Batch Size & Linear Warmup Look at $\text{Batch}_{\text{effective}} = \text{per\_device} \times \text{grad\_accum} \times \text{num\_gpus}$: - If your single GPU only has enough VRAM for **2 examples at a time**, setting `gradient_accumulation_steps = 8` waits for **8 mini-batches (16 examples total)** before taking 1 smooth weight update step — getting big-cluster training stability on a single affordable GPU! - **Warmup + Cosine Decay:** Don't jump straight to a high learning rate at Step 1 when random LoRA sheets are fresh (you'll wreck pretrained weights!). Ramp up gently (**Warmup**), then glide slowly down to zero (**Cosine Decay**) for crisp convergence!