Week 8 • Day 3Week 8

Day 3:Week 8, Day 3: `LLM-as-a-Judge` Engineering — Faithfulness Rubrics & Mitigating Judge Biases

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 8, Day 3: LLM-as-a-Judge Engineering — Faithfulness Rubrics & Mitigating Judge Biases


1. The RAG Triad Metrics (Faithfulness, Answer Relevance, Context Completeness)

When deterministic checks pass, how do we verify that the LLM didn't subtly twist a fact or invent an unsupported claim? We use LLM-as-a-Judge (Zheng et al., MT-Bench, 2023) decomposed into atomic claims:

  1. Faithfulness (Groundedness) ∈[0,1]\in [0, 1]: Break the generated response into MM atomic statements S={s1,…,sM}S = \{s_1, \dots, s_M\}. Let SsupportedS_{\text{supported}} be the subset of statements entailed by retrieved_context: Faithfulness=∣Ssupported∣∣S∣\text{Faithfulness} = \frac{|S_{\text{supported}}|}{|S|} (If a model makes 4 statements and 1 is not in the context, Faithfulness =3/4=0.75= 3/4 = 0.75, Hallucination Rate =0.25= 0.25.)

  2. Completeness (Recall vs Rubric) ∈[0,1]\in [0, 1]: What fraction of the required ground-truth facts/steps did the response cover?


2. The 3 Famous LLM-as-a-Judge Biases & How Staff Engineers Fix Them

If you naively ask an LLM "Rate this answer from 1 to 10" or "Which answer is better: A or B?", your evaluation will be corrupted by three documented biases:

Bias NameWhat Goes WrongStaff Engineering Mitigation
1. Position BiasIn pairwise A vs B comparisons, LLMs systematically favor the first option (A) ~60–70% of the time.Bidirectional Position-Swapped Judging: Run the judge twice—once as (A, B) and once swapped as (B, A). Only award a win if both orderings agree; otherwise declare a TIE!
2. Verbosity (Length) BiasLLMs reward longer, flowery responses even when they contain fluff or subtle hallucinations.Atomic Claim Decomposition + Conciseness Cap: Grade strictly on atomic claim entailment ($\frac{
3. Arbitrary Scale DriftAsking for a 1–10 or 1–100 score yields inconsistent numbers across runs (7 vs 8).Binary / 3-Point Explicit Criteria + temperature=0.0: Force the judge to output reasoning_steps before the score, using discrete criteria (0.0, 0.5, 1.0).

In Simple English: Checking Truthfulness (The Faithfulness Formula)

Look at the Faithfulness equation: Faithfulness=∣Ssupported∣∣S∣\text{Faithfulness} = \frac{|S_{\text{supported}}|}{|S|}:

  • Suppose the AI writes a 4-sentence answer (∣S∣=4|S| = 4):
    • Sentence 1 matches the runbook (Supported)
    • Sentence 2 matches the runbook (Supported)
    • Sentence 3 matches the runbook (Supported)
    • Sentence 4 invents a fake command like sudo reboot-all (Hallucinated!)
  • Faithfulness Score: $3 / 4 = \mathbf{0.75\ (75%)}$!

Defeating Judge Bias

AI judges have 2 sneaky habits: Position Bias (favoring Option A because it was printed first) and Verbosity Bias (favoring longer rambling answers). Fix both by swapping A & B twice and scoring on strict JSON rubrics!