Week 11 • Day 4Week 11

Day 4:Week 11, Day 4: Cross-Encoder Reranking & Solving the "Lost in the Middle" Phenomenon

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 11, Day 4: Cross-Encoder Reranking & Solving the "Lost in the Middle" Phenomenon


1. Stage 2 Retrieval: Cross-Encoder Reranking (Cohere Rerank, BGE-Reranker-v2)

After Stage 1 Hybrid BM25 + Dense + RRF retrieves the Top-50 candidate chunks in ~10 ms, we pass those 50 (Query, Chunk) pairs into a Cross-Encoder Reranker to select the final Top-5 chunks for the LLM prompt.

Why does a Cross-Encoder dramatically outperform RRF alone?

  • A Cross-Encoder computes full token-by-token cross-attention between the query and candidate chunk, verifying whether the query's specific relation ("how to rollback X without restarting Y") actually holds inside the chunk.

2. The "Lost in the Middle" Phenomenon (Liu et al., Stanford 2023)

Even when your Reranker finds the 5 best chunks, where you place them inside the prompt context window matters! Stanford researchers discovered a pronounced U-shaped attention curve in LLMs:

  • LLMs pay the highest attention to chunks placed at the very beginning (Position 1) and very end (Position KK, right above the question) of the prompt.
  • Information buried in the middle of a long context (Positions 3..K-2) suffers a 15–30% accuracy drop!
code
LLM Context Recall by Chunk Position ("U-Shaped Curve"): 100% │ ★ Pos 1 (Best) ★ Pos K (2nd Best) 85% │ • Pos 2 • Pos K-1 70% │ ▼ Pos 3 ▼ Pos 4 ▼ Pos 5 (Lost in the Middle!) └──────────────────────────────────────────────────────────────────────────►

Edge-Prioritized Context Reordering (Hourglass / Alternating Packing)

Given reranked chunks ordered from highest relevance [R1, R2, R3, R4, R5], we reorder them so the strongest chunks sit at the start and end of the context window:

  • R1 →\rightarrow First position (index 0)
  • R2 →\rightarrow Last position (index -1, immediately before the user question!)
  • R3 →\rightarrow Second position (index 1)
  • R4 →\rightarrow Second-to-last (index -2)
  • R5 →\rightarrow Middle! Resulting prompt order: [R1, R3, R5, R4, R2]!

In Simple English: Bi-Encoder (Fast Dragnet) + Cross-Encoder (Deep Reader) & Lost-in-the-Middle

  1. Bi-Encoder (Stage 1): Embeds the query alone and every document alone ahead of time — blazingly fast (scans millions in 5ms), good enough to grab Top 30–50 candidates.
  2. Cross-Encoder Reranker (Stage 2): Places the Question + Candidate Doc side-by-side so every word in the question attends directly to every word in the doc — way more accurate (~100ms for Top 30)!
  3. Lost-in-the-Middle Fix: Research proves LLMs pay strong attention to the very TOP and very BOTTOM of long prompts, often skimming the middle! Place your #1 best reranked doc FIRST and #2 LAST!