Day 3:Week 8, Day 3: `LLM-as-a-Judge` Engineering — Faithfulness Rubrics & Mitigating Judge Biases
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 8, Day 3: LLM-as-a-Judge Engineering — Faithfulness Rubrics & Mitigating Judge Biases
1. The RAG Triad Metrics (Faithfulness, Answer Relevance, Context Completeness)
When deterministic checks pass, how do we verify that the LLM didn't subtly twist a fact or invent an unsupported claim? We use LLM-as-a-Judge (Zheng et al., MT-Bench, 2023) decomposed into atomic claims:
-
Faithfulness (Groundedness) : Break the generated response into atomic statements . Let be the subset of statements entailed by
retrieved_context: (If a model makes 4 statements and 1 is not in the context, Faithfulness , Hallucination Rate .) -
Completeness (Recall vs Rubric) : What fraction of the required ground-truth facts/steps did the response cover?
2. The 3 Famous LLM-as-a-Judge Biases & How Staff Engineers Fix Them
If you naively ask an LLM "Rate this answer from 1 to 10" or "Which answer is better: A or B?", your evaluation will be corrupted by three documented biases:
| Bias Name | What Goes Wrong | Staff Engineering Mitigation |
|---|---|---|
| 1. Position Bias | In pairwise A vs B comparisons, LLMs systematically favor the first option (A) ~60–70% of the time. | Bidirectional Position-Swapped Judging: Run the judge twice—once as (A, B) and once swapped as (B, A). Only award a win if both orderings agree; otherwise declare a TIE! |
| 2. Verbosity (Length) Bias | LLMs reward longer, flowery responses even when they contain fluff or subtle hallucinations. | Atomic Claim Decomposition + Conciseness Cap: Grade strictly on atomic claim entailment ($\frac{ |
| 3. Arbitrary Scale Drift | Asking for a 1–10 or 1–100 score yields inconsistent numbers across runs (7 vs 8). | Binary / 3-Point Explicit Criteria + temperature=0.0: Force the judge to output reasoning_steps before the score, using discrete criteria (0.0, 0.5, 1.0). |
In Simple English: Checking Truthfulness (The Faithfulness Formula)
Look at the Faithfulness equation: :
- Suppose the AI writes a 4-sentence answer ():
- Sentence 1 matches the runbook (Supported)
- Sentence 2 matches the runbook (Supported)
- Sentence 3 matches the runbook (Supported)
- Sentence 4 invents a fake command like
sudo reboot-all(Hallucinated!)
- Faithfulness Score: $3 / 4 = \mathbf{0.75\ (75%)}$!
Defeating Judge Bias
AI judges have 2 sneaky habits: Position Bias (favoring Option A because it was printed first) and Verbosity Bias (favoring longer rambling answers). Fix both by swapping A & B twice and scoring on strict JSON rubrics!