The Lab · instrument 15/16 · interactive

Break a RAG pipeline

Real BM25 retrieval over chunks you control. The answer is always present; whether it reaches the model is a chunking decision, and watching it fail is the fastest way to learn why retrieval is the ceiling.

What's real here

The document is split at your chunk size and overlap, indexed, and scored by genuine BM25 — the keyword half of the hybrid retrieval a real pipeline runs. Everything you see is computed from the text in the box: the ranking, the scores, the context budget. Crucially, the answer is in the corpus for every question. If it doesn't reach the model, that is your pipeline's doing, not the document's.

Where to push

The defaults work: the billing-records question, chunks of 35 words, and the answer arrives. Now break it. Drag the chunk size down to 18 and switch to "who signs off on a rollback" — it misses. The document says countersigns; you said signs off; keyword scoring has no idea those mean the same thing, and the smaller chunks have thinned the surrounding context that might have rescued it. Now add six words of overlap and watch it land. That single flip is the reason overlap exists, and it took no model change, no better prompt, and no bigger context — just a chunking parameter almost nobody tunes.

Then drive the size slider to its extremes. Tiny chunks fragment the answer and dilute the signal; enormous chunks retrieve everything and spend your entire context budget on one hit — watch the token count climb as you drag. Somewhere in between is the setting that works for your documents, and the only way to find it is to measure, which is exactly what this page is doing.

Failure mode

Tuning chunk size on one question. This page gives you six, and they disagree — the setting that rescues the cross-reference question is not the setting that keeps the definition intact. Teams do the same thing with a single failing report from a colleague, move the slider until that one works, and ship a configuration nobody measured against anything else. Chunking is a parameter like any other: it needs a set of real questions with known answers, and a recall number you can watch move. Otherwise you are not tuning, you are alternating which question is broken. (The other reflex — blaming the model when the verdict line turns violet — is the one the pipeline entry exists to correct.)