Week 8 • Day 5Week 8

Day 5:Week 8, Days 5–7: Month 2 Capstone — Automated LLM Evaluation & Regression Test Suite

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 8, Days 5–7: Month 2 Capstone — Automated LLM Evaluation & Regression Test Suite


1. Capstone Architecture (capstones/month02_automated_llm_eval_suite/)

In Month 1, we built the SRE Runbook Copilot CLI (capstones/month01_sre_runbook_copilot_cli/). Now, in Month 2 Capstone, we wrap our SRE Copilot in a production-grade Automated LLM Evaluation & CI/CD Regression Harness:

code
capstones/month02_automated_llm_eval_suite/ ├── README.md ├── evals/ │ ├── golden_sre_dataset.json # Stratified Golden Dataset (Happy Path, Multi-Hop, Out-of-Scope, Adversarial) │ └── baseline_metrics.json # Frozen Production Baseline Metrics ├── .github_workflow_evals.yml # Reference GitHub Actions CI/CD Gate Workflow ├── src/ │ └── eval_runner.py # End-to-End Evaluation Harness, Judge Rubric, Confidence Intervals & Markdown Report Generator └── tests/ └── test_capstone_month02.py # Automated Verification Suite

In Simple English: Your Automated AI Quality Gate (Month 2 Capstone)

Imagine shipping a new version of an airplane autopilot without test flights — nobody would fly on it! Yet most teams deploy new prompts and AI models with zero automated tests. Your Month 2 Capstone Automated Eval Suite acts like an automated CI/CD quality gate:

  1. Runs every question in your stratified Golden Dataset.
  2. Executes Layer 1 Deterministic Checks (valid JSON structure, zero leaked secrets, valid source citations).
  3. Computes Claim Faithfulness Score & Abstention Accuracy (did it say "I don't know" on unanswerable traps?).
  4. Checks 95% Confidence Intervals before returning exit code 0 (Deploy!) or 1 (Block Release!).

Try It Free in Your Browser: Open Tab 2 (Codes) and Tab 4 (Test & Troubleshooting) — we added a self-contained runnable Python Eval Gate (eval_suite_runner.py) and unittest verification suite so you can execute the capstone logic directly inside your browser!