RAG & Embeddings · entry 05/06 · 3 min read

Evaluating retrieval

A RAG system has two failure surfaces and one of them sets the limit — how to score retrieval separately, what to score the answer on, and the eval set you have to build yourself.

Two systems, measured separately

A grounded answer can fail two ways: the evidence never arrived, or the model mishandled evidence that did. Measuring only the final answer tells you something failed and nothing about which — and the two have different fixes, different owners, and wildly different costs. So score the retriever alone first. Its number is the ceiling on grounded answers: no prompt, model, or amount of context can recover an answer whose evidence was never in the window. The model may still answer correctly from what it memorized in pretraining — which is why end-to-end accuracy above your recall number is a warning rather than headroom. Those answers are unattributable, quietly stale, and absent on the private half of your corpus.

Retrieval metrics that mean something

Hit rate@k — how often at least one answer-bearing passage lands in the k you retrieve — is the one that matters most, because it sets that ceiling. (It coincides with recall@k when there is exactly one gold passage; for multi-hop questions, label the whole set of passages the answer needs and measure what fraction arrived.) Rank matters too, since evidence at position twenty competes with nineteen distractors: MRR or nDCG capture whether the right passage arrives near the top. Measure at both k's — the candidate depth you retrieve before reranking, and the k you actually send. A gap between the two belongs to the reranker; a low number at both belongs to retrieval. And re-measure after index churn, since approximate indexes decay as they absorb inserts and deletes.

Generation metrics, once the evidence is there

Given retrieved passages, two questions remain. Faithfulness: is every claim in the answer supported by those passages? Answer relevance: does it address what was asked? Both are open-ended, so both are usually scored by an LLM judge against the retrieved text — with all the care that requires, calibration against human labels included. Report them conditional on retrieval succeeding; mixed together, a retrieval regression masquerades as a generation one.

The eval set is the work

Nobody can give you this: it has to come from your corpus. Take real questions — from logs, from support tickets, from the people who will use it — and for each, record the passage that answers it. A hundred carefully labeled pairs beat ten thousand synthetic ones, and models can help draft candidate questions from passages as long as a human confirms them. Refresh it as the corpus changes, and keep the questions that once failed in production in it permanently.

Failure mode

Evaluating end-to-end only, then tuning the prompt. The number moves a little, everyone feels productive, and the actual constraint — recall at your k — never changes, because nothing anyone did touched it. Measure the two surfaces separately before you optimize either, and when the ceiling is the problem, the fixes are chunking, hybrid search and reranking, not wording.