Day 5:Week 8, Days 5–7: Month 2 Capstone — Automated LLM Evaluation & Regression Test Suite
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 8, Days 5–7: Month 2 Capstone — Automated LLM Evaluation & Regression Test Suite
1. Capstone Architecture (capstones/month02_automated_llm_eval_suite/)
In Month 1, we built the SRE Runbook Copilot CLI (capstones/month01_sre_runbook_copilot_cli/).
Now, in Month 2 Capstone, we wrap our SRE Copilot in a production-grade Automated LLM Evaluation & CI/CD Regression Harness:
capstones/month02_automated_llm_eval_suite/
├── README.md
├── evals/
│ ├── golden_sre_dataset.json # Stratified Golden Dataset (Happy Path, Multi-Hop, Out-of-Scope, Adversarial)
│ └── baseline_metrics.json # Frozen Production Baseline Metrics
├── .github_workflow_evals.yml # Reference GitHub Actions CI/CD Gate Workflow
├── src/
│ └── eval_runner.py # End-to-End Evaluation Harness, Judge Rubric, Confidence Intervals & Markdown Report Generator
└── tests/
└── test_capstone_month02.py # Automated Verification Suite
In Simple English: Your Automated AI Quality Gate (Month 2 Capstone)
Imagine shipping a new version of an airplane autopilot without test flights — nobody would fly on it! Yet most teams deploy new prompts and AI models with zero automated tests. Your Month 2 Capstone Automated Eval Suite acts like an automated CI/CD quality gate:
- Runs every question in your stratified Golden Dataset.
- Executes Layer 1 Deterministic Checks (valid JSON structure, zero leaked secrets, valid source citations).
- Computes Claim Faithfulness Score & Abstention Accuracy (did it say "I don't know" on unanswerable traps?).
- Checks 95% Confidence Intervals before returning exit code
0(Deploy!) or1(Block Release!).
Try It Free in Your Browser: Open Tab 2 (Codes) and Tab 4 (Test & Troubleshooting) — we added a self-contained runnable Python Eval Gate (
eval_suite_runner.py) andunittestverification suite so you can execute the capstone logic directly inside your browser!