Prompt Engineering · entry 06/07 · 3 min read

Optimizing prompts

If you have an eval, a prompt is a parameter you can search rather than a sentence you agonize over — automatic optimization, what it buys, and the ways it quietly overfits.

Hand-tuning has a ceiling

The usual workflow is a person rewording a prompt, eyeballing a few outputs, and stopping when it looks better. That is overfitting to a sample of one, performed manually, and it does not survive a model upgrade. The alternative needs one prerequisite: a graded eval set. With that, the prompt stops being prose you argue about and becomes a parameter you can search — and search does not get bored, does not have a favorite phrasing, and does not stop early because the third example looked fine.

What can actually be searched

Two things reliably, and a third that is still research. Instruction wording: propose variants — with a model, since it is good at paraphrase — score each on the eval, keep the winner, repeat. Examples: which few-shot cases to include and in what order, with the strongest frameworks generating them by running your program and keeping the traces that passed the metric — including for intermediate steps that have no labels at all. Which of the two matters more is task-dependent — examples dominate on format and pattern-matching work, instructions where the rules are conditional — and optimizing both together is the safe default. Pipeline structure — how many steps, what each outputs — is the third, and today you mostly still design it by hand; automated structure search exists but is a newer research line. The useful mental model is compilation: you specify what the program should do and what counts as good, and the optimizer searches for the prompt that scores best — with the caveat that a compiler proves and a search only fits.

Where it earns and where it doesn't

It earns when the task is stable, the eval is honest, and volume justifies the setup — classification, extraction, routing, anything run thousands of times. It does not earn on one-off requests or where you have no way to grade the answer, which is most exploratory work. And it changes the maintenance story: you version the eval and the optimizer configuration alongside the compiled prompt — the search is stochastic, so unlike a real build the output cannot be regenerated from its inputs, and a prompt you didn't check in cannot be diffed or rolled back. A model swap then means re-running the search rather than re-negotiating paragraphs.

Failure mode

Optimizing against an eval too small to carry the weight. Search is relentless and will find whatever quirk your fifty examples contain — a phrasing that games one grader, an instruction that helps exactly those cases. The result looks superb on the set it was fitted to and mediocre in production, which is the oldest failure in machine learning, now automated. Hold out cases the optimizer never sees and keep them fresh — and if your metric is a model judge, size will not save you: an optimizer is a Goodhart engine pointed at your grader, so keep the unoptimized holdout and re-calibrate the judge whenever the score jumps. Automatic optimization raises the value of your eval and the cost of a bad one at the same time.