Week 8 • Day 2Week 8

Day 2:Week 8, Day 2: Deterministic & Heuristic Evals — Schema, Citation, Keyword & Safety Assertions

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 8, Day 2: Deterministic & Heuristic Evals — Schema, Citation, Keyword & Safety Assertions


1. The Evaluation Pyramid: Run Fast Deterministic Checks First!

Many engineers mistakenly use an expensive LLM-as-a-Judge call to check things that a 0.01 ms Python function can verify with 100% mathematical certainty!

code
/\ / \ Layer 3: Human Expert Review (Slow, $$$, Gold Standard) /────\ / \ Layer 2: LLM-as-a-Judge Rubrics (~500ms, $, Semantic Nuance) /────────\ / \ Layer 1: Deterministic Assertions (<1ms, $0, 100% Exact!) /────────────\ - JSON/Pydantic Schema Validity - Citation Grounding (Are cited docs actually in context?) - Required Fact Recall & Forbidden Command Regex Gate - Abstention Verification

2. The 4 Mandatory Deterministic Evaluators

  1. Fact Recall Score (Rfacts∈[0,1]R_{\text{facts}} \in [0, 1]): Rfacts=∣{f∈ExpectedFacts:normalize(f)⊆normalize(Output)}∣∣ExpectedFacts∣R_{\text{facts}} = \frac{|\{f \in \text{ExpectedFacts} : \text{normalize}(f) \subseteq \text{normalize}(\text{Output})\}|}{|\text{ExpectedFacts}|}
  2. Citation Precision & Grounding (Gcite∈[0,1]G_{\text{cite}} \in [0, 1]): Every citation [doc.md] emitted by the model must exist in the provided context (No Phantom Citations) and cover expected_citations.
  3. Safety / Forbidden Pattern Gate (Ssafe∈{0,1}S_{\text{safe}} \in \{0, 1\}): Zero tolerance: if any forbidden_patterns (e.g., rm -rf /, DROP DATABASE, leaked API keys) match the output, score is strictly 0.0.
  4. Abstention Accuracy (Aabstain∈{0,1}A_{\text{abstain}} \in \{0, 1\}): If must_abstain=True, the model must explicitly flag abstained=True or state insufficient context rather than fabricating steps.

In Simple English: Cheap Fast Guardrails BEFORE Expensive AI Judges

Think of automated grading like airport security:

  1. Layer 1 — Deterministic Checks (Metal Detector: Free, 0.1ms): Does the output parse as valid JSON? Are all required fields present? Does it leak any sk-... secret key or profanity? If Layer 1 fails, reject immediately at $0.00 cost!
  2. Layer 2 — LLM-as-a-Judge (Human Screener: ~500ms, low cost): Check softer meaning questions like "Did the assistant actually answer what the user asked cleanly?"

Always run 100% of deterministic checks first in CI/CD before paying a single cent for judge model calls!