Week 11 • Day 4Week 11
Day 4:Week 11, Day 4: Cross-Encoder Reranking & Solving the "Lost in the Middle" Phenomenon
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 11, Day 4: Cross-Encoder Reranking & Solving the "Lost in the Middle" Phenomenon
1. Stage 2 Retrieval: Cross-Encoder Reranking (Cohere Rerank, BGE-Reranker-v2)
After Stage 1 Hybrid BM25 + Dense + RRF retrieves the Top-50 candidate chunks in ~10 ms, we pass those 50 (Query, Chunk) pairs into a Cross-Encoder Reranker to select the final Top-5 chunks for the LLM prompt.
Why does a Cross-Encoder dramatically outperform RRF alone?
- A Cross-Encoder computes full token-by-token cross-attention between the query and candidate chunk, verifying whether the query's specific relation ("how to rollback X without restarting Y") actually holds inside the chunk.
2. The "Lost in the Middle" Phenomenon (Liu et al., Stanford 2023)
Even when your Reranker finds the 5 best chunks, where you place them inside the prompt context window matters! Stanford researchers discovered a pronounced U-shaped attention curve in LLMs:
- LLMs pay the highest attention to chunks placed at the very beginning (Position 1) and very end (Position , right above the question) of the prompt.
- Information buried in the middle of a long context (
Positions 3..K-2) suffers a15–30%accuracy drop!
code
LLM Context Recall by Chunk Position ("U-Shaped Curve"):
100% │ ★ Pos 1 (Best) ★ Pos K (2nd Best)
85% │ • Pos 2 • Pos K-1
70% │ ▼ Pos 3 ▼ Pos 4 ▼ Pos 5 (Lost in the Middle!)
└──────────────────────────────────────────────────────────────────────────►
Edge-Prioritized Context Reordering (Hourglass / Alternating Packing)
Given reranked chunks ordered from highest relevance [R1, R2, R3, R4, R5], we reorder them so the strongest chunks sit at the start and end of the context window:
R1First position (index 0)R2Last position (index -1, immediately before the user question!)R3Second position (index 1)R4Second-to-last (index -2)R5Middle! Resulting prompt order:[R1, R3, R5, R4, R2]!
In Simple English: Bi-Encoder (Fast Dragnet) + Cross-Encoder (Deep Reader) & Lost-in-the-Middle
- Bi-Encoder (Stage 1): Embeds the query alone and every document alone ahead of time — blazingly fast (scans millions in 5ms), good enough to grab Top 30–50 candidates.
- Cross-Encoder Reranker (Stage 2): Places the Question + Candidate Doc side-by-side so every word in the question attends directly to every word in the doc — way more accurate (~100ms for Top 30)!
- Lost-in-the-Middle Fix: Research proves LLMs pay strong attention to the very TOP and very BOTTOM of long prompts, often skimming the middle! Place your #1 best reranked doc FIRST and #2 LAST!