Week 8 • Day 2Week 8
Day 2:Week 8, Day 2: Deterministic & Heuristic Evals — Schema, Citation, Keyword & Safety Assertions
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 8, Day 2: Deterministic & Heuristic Evals — Schema, Citation, Keyword & Safety Assertions
1. The Evaluation Pyramid: Run Fast Deterministic Checks First!
Many engineers mistakenly use an expensive LLM-as-a-Judge call to check things that a 0.01 ms Python function can verify with 100% mathematical certainty!
code
/\
/ \ Layer 3: Human Expert Review (Slow, $$$, Gold Standard)
/────\
/ \ Layer 2: LLM-as-a-Judge Rubrics (~500ms, $, Semantic Nuance)
/────────\
/ \ Layer 1: Deterministic Assertions (<1ms, $0, 100% Exact!)
/────────────\ - JSON/Pydantic Schema Validity
- Citation Grounding (Are cited docs actually in context?)
- Required Fact Recall & Forbidden Command Regex Gate
- Abstention Verification
2. The 4 Mandatory Deterministic Evaluators
- Fact Recall Score ():
- Citation Precision & Grounding ():
Every citation
[doc.md]emitted by the model must exist in the provided context (No Phantom Citations) and coverexpected_citations. - Safety / Forbidden Pattern Gate ():
Zero tolerance: if any
forbidden_patterns(e.g.,rm -rf /,DROP DATABASE, leaked API keys) match the output, score is strictly0.0. - Abstention Accuracy ():
If
must_abstain=True, the model must explicitly flagabstained=Trueor state insufficient context rather than fabricating steps.
In Simple English: Cheap Fast Guardrails BEFORE Expensive AI Judges
Think of automated grading like airport security:
- Layer 1 — Deterministic Checks (Metal Detector: Free, 0.1ms): Does the output parse as valid JSON? Are all required fields present? Does it leak any
sk-...secret key or profanity? If Layer 1 fails, reject immediately at $0.00 cost! - Layer 2 — LLM-as-a-Judge (Human Screener: ~500ms, low cost): Check softer meaning questions like "Did the assistant actually answer what the user asked cleanly?"
Always run 100% of deterministic checks first in CI/CD before paying a single cent for judge model calls!