Large Language Models · entry 05/06

Hallucination

Models are trained to produce plausible text, and truth is only correlated with plausible — hallucination is the gap, and it is managed, not cured.

Why it happens

Pretraining rewards the most likely continuation, not the most true one. For well-attested facts, likely and true coincide; at the edges — obscure citations, niche APIs, people with thin footprints — the model still must emit something, and what it emits is shaped like a right answer: real journal names around an invented paper, plausible arguments in a fabricated case. The polish isn't deception; it's the training objective doing exactly what it was asked.

Where it concentrates

Risk clusters predictably: specifics (numbers, dates, quotes, URLs, citations) over generalities; the long tail over the well-documented; anything after the training cutoff; and requests where refusing feels unhelpful — preference training taught models that a confident attempt often scores better than "I don't know," which is the sycophancy problem wearing a lab coat.

What actually reduces it

Grounding: put the truth in the context — retrieval, tool results, pasted documents — and instruct answers from the provided material with citations. Models are far better at reading than remembering. Room to abstain: explicitly making "not in the documents" an acceptable answer measurably cuts fabrication. Verification: check high-stakes outputs against sources, run generated code, resolve cited URLs; second-pass self-checking ("verify each claim above against the text") catches a real fraction. None of these reach zero.

Design for it

Treat model output like a bright junior colleague's draft: excellent recall of the gist, unreliable on citations, never to be shipped unreviewed where money, health, law or reputation ride on it. Product design does heavy lifting: show sources, expose uncertainty, make verification one click instead of an act of discipline.

Failure mode

Trust calibrated by fluency. Nine correct answers build a habit of believing the tenth, and the tenth arrives in the same confident prose — the operator error isn't using the model, it's letting tone stand in for evidence. Fluency is constant; accuracy varies by distance from the training distribution. Calibrate on that.