Machine Learning Foundations · entry 07/10 · 3 min read

Learning by building

Six small projects, in the order that teaches most — and the reason a year of reading leaves you unable to recognize the failures you can define perfectly.

Why reading plateaus

You can read this entire site and still not recognize overfitting in your own loss curves, still not notice that your eval set leaked, still not feel the moment a prompt change helped one case and broke four. That is not a failure of the reading. Vocabulary and judgment are different faculties, and only one of them is transferable by text: judgment comes from having been wrong about something specific and having had to find out why.

Six projects, in order

Each of these teaches one thing that reading cannot, and each takes an evening rather than a month.

1  train a small classifier on a tabular dataset, end to end
   — including the split, a naive baseline, and a real metric
2  overfit it deliberately, then fix it
3  script an LLM call with a rubric and thirty of your own test cases
4  build a tiny retrieval system over documents you actually own
5  give that system one tool and a loop, and let it run unattended
6  reimplement one mechanism from scratch: a tokenizer, or attention,
   or gradient descent, in a hundred lines and no framework

The order is not arbitrary. One and two install the reflex that a number is a claim about a specific set of data. Three is where most people first discover that a prompt improvement they were sure about is noise. Four teaches that retrieval quality dominates long before model quality does. Five is where step-level accuracy meets multiplication. Six is what stops the mechanism being magic — and it is worth doing exactly once, on whichever piece still feels like a black box.

The habit that makes it stick

Before each run, write down what you expect to happen and why. After it, write what actually happened. The value is entirely in the gap: a prediction you got wrong is the only reliable signal that a belief of yours was wrong, and without the written record you will quietly remember having expected whatever occurred. This is the same discipline that makes an eval set useful, applied to yourself.

Failure mode

Mistaking tutorial completion for skill. A good tutorial removes exactly the parts that teach — it hands you a clean dataset, a defined label, a chosen metric, and a split someone else made, which are four of the five decisions that determine whether real work succeeds. Finishing it feels like learning and produces almost none. Do the tutorial if you need the API surface, then immediately redo it on data you found yourself, where nobody has told you what the right answer looks like. That second version is the one that teaches. The Lab on this site exists for the same reason: reading about a decision boundary and bending one with your own mouse are not the same experience.