RAG & Embeddings · entry 04/04

RAG vs fine-tuning

Retrieval changes what the model can see; fine-tuning changes how it behaves — and most teams that think they need the second need the first.

Two levers, usually confused

Both get sold as "customizing the model," and that framing causes most of the bad decisions. They act on different parts of the system. Retrieval changes the input: fresh documents placed in context at query time, weights untouched. Fine-tuning changes the model itself: additional training on your examples until its default behavior shifts. One hands the model a library card; the other sends it to finishing school.

Knowledge wants retrieval

Facts that change, facts that are private, and facts that need an audit trail all point the same way. With RAG, updating knowledge is re-indexing a document — minutes, no training run. Access control works per user, at query time: retrieval simply never returns documents this user cannot read, something weights cannot do, because a model cannot selectively forget for one audience. Answers carry citations. And fine-tuning is genuinely bad at this job: a model tuned on a few thousand pages does not reliably memorize them — it absorbs their vocabulary and their confidence, then improvises the details.

Behavior wants training

Fine-tuning earns its cost when the problem is form rather than fact: holding a strict output schema, writing in the house voice, using a specialty's vocabulary correctly, refusing what your policy refuses. Tuned weights apply on every response without spending context tokens on instructions. But climb the cheap ladder first — few-shot examples and structured output solve most format-and-style problems with no training pipeline, no eval harness, and no redeploy when requirements drift next quarter.

Costs, cadence, composition

RAG pays at query time: retrieval infrastructure plus context tokens on every request, updated in minutes. Fine-tuning pays up front: data curation, training runs, evaluation, and a repeat of all three on every change — which is why tuning is reserved for what stays stable. The two compose cleanly, and strong systems often do both: a model tuned for form, fed today's facts by a retrieval pipeline.

Facts change, or differ by user?     -> RAG
Need citations or access control?    -> RAG
Format or voice keeps drifting?      -> prompt first, then fine-tune
Domain language the model mangles?   -> fine-tune
Both problems at once?               -> tune for form, retrieve facts

Failure mode

Fine-tuning in a knowledge base. A team pours three years of internal documents into a training run to "teach the model the company." The result sounds exactly right — same headings, same acronyms — and that fluency passes review until the model invents a policy in the house voice, with no citation to check. Updating anything now means another training run; deleting anything is nearly impossible; per-user access control never existed. Fresh, checkable, revocable knowledge is the job description of retrieval. Use training for behavior, and keep facts where you can see them.