RAG & Embeddings · entry 03/04
The RAG pipeline
Chunk the documents, embed them, retrieve what matches the question, and hand the model evidence with sources — the standard architecture for grounded answers.
Why bolt retrieval onto a model
A model knows what its training data contained, as of its cutoff — nothing about your wiki, last week's incident, or this customer's contract. Retrieval-augmented generation fixes that at answer time: find the few passages relevant to the question, place them in the context window, and have the model answer from that evidence. Grounded this way, models hallucinate measurably less, answers can carry citations a reader can check, and updating the system's knowledge means updating an index — not the model.
Chunking
Documents are too long to embed whole, so they are split into chunks — and chunking is a design decision, not plumbing. A few hundred tokens per chunk is the usual sweet spot: small enough that one vector means one thing, large enough to read on its own. Overlap adjacent chunks slightly so a fact straddling a boundary survives intact somewhere. Respect structure — split at headings and paragraph breaks, keep tables and code blocks whole — and attach metadata (source, title, section) to every chunk, because citations and filters will need it later.
Retrieve, rerank
At query time the pipeline embeds the question and pulls the nearest chunks by vector search. The strong default is hybrid retrieval: run BM25 keyword search in parallel — vectors fumble part numbers and function names; keywords catch them — and merge the two lists with reciprocal rank fusion. Then rerank: a cross-encoder that reads query and chunk together orders candidates far better than raw vector distance. Retrieve fifty, rerank, keep five.
Assemble and answer
The final prompt is ordinary prompt anatomy with the evidence in the middle:
Answer using ONLY the sources below. Cite as [1], [2].
If the sources do not contain the answer, say so.
[1] deploy-guide.md, "Rollbacks": ...chunk text...
[2] runbook.md, "On-call": ...chunk text...
Question: {question}The "say so" escape hatch is load-bearing: without it, a model handed irrelevant sources answers anyway. Numbered source tags cost almost nothing and make every claim checkable.
Failure mode
Blaming the model for the retriever's sins. Retrieval quality dominates the whole system: if the answer-bearing chunk never reaches the prompt, the best model available either refuses or improvises — a good model fed garbage produces fluent garbage. Teams see wrong answers and swap models or polish prompts when the fix was recall. Evaluate retrieval on its own: for a set of real questions, how often is the right chunk in the top k? That number is usually the system's ceiling; raise it before touching anything else.