RAG & Embeddings · entry 04/05

Agentic retrieval

One-shot retrieval answers one-hop questions. For everything else, the model plans searches, reads, follows leads and searches again — research as a loop, with a bill to match.

The ceiling of one-shot RAG

The classic pipeline embeds the question, grabs the top chunks, and answers — one hop, one chance. Real questions are rarely so cooperative: "did our refund policy change between the 2024 and 2026 contracts, and which customers does that affect?" needs multiple searches, comparison across documents, and a join no single chunk contains. When the answer is scattered, the top-k for the original phrasing of the question is the wrong k chunks, and no amount of reranking fixes a question that needed to become three questions.

Retrieval as a tool in a loop

The agentic version hands search to the model as a tool: it decomposes the question, issues queries it invents (often better keyword queries than the user's prose), reads results, notices what's missing, reformulates, and stops when it has enough — the agent loop pointed at a corpus. Deep-research products layer planning and citation on top: minutes of autonomous searching that return a sourced report. The same pattern quietly fixes classic RAG failures — acronyms get expanded, dates get constrained, and a dead-end search becomes a lesson instead of an answer.

Structure when relationships matter

Some questions are about connections — who approved what, which services depend on the failing one — and similarity search over prose is a blunt tool for edges. Knowledge-graph flavors of retrieval extract entities and relations at index time so multi-hop questions traverse links instead of hoping a chunk mentions both ends; hybrid systems let the agent walk the graph and read the source passages it anchors to. It earns its indexing cost where relationship questions recur — org charts, dependencies, litigation timelines — and is over-engineering where they don't.

Failure mode

Paying agent prices for lookup questions. The loop multiplies everything: five searches, three rereads, one sub-agent — ten times the tokens and latency of one retrieval — and for "what's our PTO policy," the pipeline's answer was identical and instant. Route by question shape: single-hop lookups to the cheap path, genuine research to the loop, and let an eval on your real traffic — not the demo's hardest question — set the split.