Week 11, Day 6: Information Retrieval Metrics Math — Recall@K, Precision@K, MRR & nDCG@K

1. The Four Mandatory Information Retrieval (IR) Metrics

Before touching the Generator LLM, how do you mathematically prove whether Hybrid + Reranker is better than Dense-Only? You evaluate the Retriever on four classic IR metrics at cutoff $K$:

1. Recall@K (Did we find the needle in the Top-$K$?)

Given ground-truth relevant document set $G$ and Top-$K$ retrieved list $R_K = [d_1, \dots, d_K]$: $\text{Recall@K} = \frac{|R_K \cap G|}{|G|}$

2. Precision@K (What fraction of the Top-$K$ is relevant?)

$\text{Precision@K} = \frac{|R_K \cap G|}{K}$

3. MRR (Mean Reciprocal Rank — How high was the first relevant hit?)

If the highest-ranked relevant document for query $q$ appears at 1-based rank $\text{rank}_q^*$: $\text{RR}_q = \frac{1}{\text{rank}_q^*}, \qquad \text{MRR} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\text{rank}_q^*}$ (Rank 1 = 1.0, Rank 2 = 0.5, Rank 3 = 0.333, Not found = 0.0.)

2. nDCG@K (Normalized Discounted Cumulative Gain — Graded Relevance & Rank Discounting!)

Recall@K treats Rank 1 and Rank K identically, and MRR only looks at the first relevant result. What if Query $q$ has multiple relevant chunks with graded relevance $\text{rel}_i \in \{0, 1, 2, 3\}$ (e.g., 3 = exact runbook command, 1 = background architecture overview, 0 = irrelevant)?

1. Discounted Cumulative Gain (DCG@K): Discounts lower ranks logarithmically by $\frac{1}{\log_2(i + 1)}$: $\text{DCG@K} = \sum_{i=1}^{K} \frac{2^{\text{rel}_i} - 1}{\log_2(i + 1)}$
2. Ideal DCG (IDCG@K): The maximum possible DCG@K achieved by sorting the ground-truth relevant items in perfect descending order of $\text{rel}_i$.
3. Normalized DCG (nDCG@K $\in [0, 1]$): $\text{nDCG@K} = \frac{\text{DCG@K}}{\text{IDCG@K}}$ An nDCG@K of 1.0 means the retriever placed every relevant document in the exact ideal order at the very top of the list!

In Simple English: Grading Search Results (Recall@K, MRR, nDCG@K)

Suppose 2 truly relevant docs exist for a user question, and your search engine returns Top 5 results (K = 5):

• Recall@5 ("Did we find the needles?"): If 1 of the 2 relevant docs appeared in Top 5 $\rightarrow$ Recall@5 = $1/2 = \mathbf{50%}$. (Most critical RAG metric — if the right doc isn't retrieved, the LLM CANNOT answer!)
• MRR ("How high was the FIRST right answer?"): If the first right doc landed at Rank #1 $\to 1/1 = 1.0$; at Rank #2 $\to 1/2 = 0.5$; at Rank #5 $\to 1/5 = 0.2$.
• nDCG@K ("Graded 0–3 star quality with top-slot bonus"): Rewards putting a 3-star exact match at Rank #1 instead of burying it down at Rank #5!