Week 8 Master Guide
Week 8: Automated LLM Evaluation, `LLM-as-a-Judge` & CI/CD Regression Gates (+ Month 2 Capstone)
Week 8 Daily Breakdown (5 Days)
Day 1
Week 8, Day 1: Why "Vibes" Fail — Designing Stratified Golden Evaluation Datasets
1 code file(s) • 3 Q&ADay 2
Week 8, Day 2: Deterministic & Heuristic Evals — Schema, Citation, Keyword & Safety Assertions
1 code file(s) • 3 Q&ADay 3
Week 8, Day 3: `LLM-as-a-Judge` Engineering — Faithfulness Rubrics & Mitigating Judge Biases
1 code file(s) • 3 Q&ADay 4
Week 8, Day 4: Statistical Significance, Confidence Intervals & CI/CD Regression Gates
1 code file(s) • 3 Q&ADay 5
Week 8, Days 5–7: Month 2 Capstone — Automated LLM Evaluation & Regression Test Suite
1 code file(s) • 0 Q&AWeek 8: Automated LLM Evaluation, LLM-as-a-Judge & CI/CD Regression Gates (+ Month 2 Capstone)
Month 2 — Classical ML, Deep Learning, Transformers & LLM Evaluation Goal: Replace manual "vibe-checking" with a deterministic +
LLM-as-a-Judgeevaluation harness that blocks regressions in CI/CD before any prompt or model change reaches production.
Weekly Architecture & Daily Breakdown
- Day 1: Why "Vibes" Fail — Designing Stratified Golden Evaluation Datasets (
day01_why_vibes_fail_golden_datasets_and_eval_taxonomies/) - Day 2: Deterministic & Heuristic Evals — Schema, Citation, Keyword & Safety Assertions (
day02_deterministic_assertion_evals_and_schema_metrics/) - Day 3:
LLM-as-a-JudgeEngineering — Faithfulness Rubrics & Mitigating Judge Biases (day03_llm_as_a_judge_rubrics_and_bias_mitigation/) - Day 4: Statistical Significance, Confidence Intervals & CI/CD Regression Gates (
day04_statistical_eval_confidence_intervals_and_ci_cd_gates/) - Days 5–7 (Month 2 Capstone): Automated LLM Evaluation & Regression Test Suite (
day05_to_07_month2_capstone_automated_eval_suite/&capstones/month02_automated_llm_eval_suite/)