section 03 · status: live · 11 entries · updated 2026-08-29
Large Language Models
The technology that pulled AI into everyday life. What a token is, how attention works, how a raw internet-scale model becomes an assistant that reasons in more than one modality, why it hallucinates, and what scale does and doesn't buy.
live · attention weighing the context, then sampling the next token
- Transformers & attention2 minevery token weighs everything before it
- Tokens & tokenization2 minthe atoms of model perception
- How LLMs are trained2 minfrom internet prediction to instruction following
- Fine-tuning in practice2 mina thousand good examples and an adapter
- Models in other languages3 minthe same model is a different product in Thai
- Context windows & memory2 minthe model's whole world, measured in tokens
- Hallucination2 minfluent, confident, and wrong by design
- Reasoning models3 minbuying accuracy with thinking time
- One model, many senses3 minbolted-on sight became built-in perception
- Beyond the dense transformer3 minnot every parameter has to work on every token
- Scaling laws & small models2 minpredictable gains, surprising arrivals, shrinking costs
check yourselfAnswer before you open
Trying to recall something teaches it better than re-reading does. Have a go, then open the answer.
Why do language models struggle to count the letters in a word?
They usually never see letters. Text arrives as tokens — 'strawberry' may be three opaque ids — so spelling, rhyming and character arithmetic are performed through a blindfold woven at tokenization time. Tokens & tokenization →
A model invents a plausible-looking citation. Is this a bug?
No — it is the training objective working. Pretraining rewards the most likely continuation, not the most true one, and at the edges of what it knows the likely-shaped answer is a fabrication. Grounding, permission to abstain, and verification reduce it; nothing reaches zero. Hallucination →
Someone claims their 400B model beats a 70B model. What do you ask?
Whether the 400B is sparse. In a mixture-of-experts model only a few billion parameters are active per token, so total parameters stop being a unit of comparison — you want active parameters, total parameters, and a result on your own eval. Beyond the dense transformer →
Your prompt works in English and fails in Turkish. Name three independent causes.
Data — pretraining corpora are overwhelmingly English, so the model simply knows less. Tokenization — the vocabulary was fit to English, so the same sentence costs more tokens and fits less context. Alignment — safety and instruction tuning were done mostly in English, so refusals and formatting behave differently. They need different fixes. Models in other languages →
Your assistant 'remembers' a conversation from last week. What is actually happening?
Nothing inside the model. Weights do not change between requests; the model sees only what is in its context window right now. Something stored last week's text and put it back in front of the model — which means memory is a retrieval and curation problem, with all the staleness and relevance failures that implies. Context windows & memory →
A model reads your photographed receipt beautifully in the demo and badly in production. Where do you look?
At the modality, not the model. Marketing sells one system, but perception degrades on resolution, lighting, accent and noise regardless of whether the senses were bolted on or trained in — and pipelines often downscale your image before the encoder sees it. Build a small eval per modality on your own inputs; a single headline number hides the weakest one. One model, many senses →