Prompt Engineering · entry 01/04

Anatomy of a prompt

A prompt is four parts — instructions, context, constraints, format — and where each part sits changes both accuracy and your bill.

Two channels, different jobs

Chat APIs give you two slots for words: a system message and user messages. The split is not cosmetic — models are post-trained to read system text as standing policy: who the model is, what rules bind it, which tools exist. The user turn carries the task of the moment and the material it operates on. Put per-request data in the system slot and you defeat caching; put policy in the user slot and it competes with whatever untrusted text got pasted in alongside it.

The four working parts

Nearly every production prompt decomposes into instructions (what to do), context (what to work on), constraints (scope, length, what never to do), and output format (the shape of the answer). Keep the parts visibly separate — labeled sections or XML-style tags beat one run-on paragraph, because delimiters tell the model where quoted material stops and your voice resumes:

SYSTEM
You are a support engineer for Acme's REST API.
Answer only from the provided documentation.
If the docs do not cover it, say so.

USER
<docs> ...retrieved passages... </docs>
<question>Why does POST /v2/orders return 429?</question>
Answer in three sentences, then cite the doc section.

Position is part of the prompt

Attention over a long context is not uniform. Models weight the beginning and end of a prompt more reliably than the middle — the well-replicated "lost in the middle" effect — so an instruction buried under forty thousand tokens of pasted logs gets skipped, not defied. Instructions and constraints go early, bulk context in the middle, and when the context runs long, restate the output format in one line at the very end. A bigger context window does not repeal any of this; it just gives you more middle to lose things in.

Stable prefix first

Order also has a billing consequence. Providers cache the computed state of a prompt prefix and discount matching tokens steeply — commonly to a tenth of list price — but only while the prefix matches a prior request exactly, from position zero. That turns ordering into an engineering decision: system rules, tool definitions, and few-shot examples — everything that never changes — go first, in a fixed order; the volatile parts (the user's question, retrieved documents, session state) go last. A timestamp helpfully placed at the top of the system message invalidates the entire cache on every call.

Failure mode

The kitchen-sink prompt. Every incident adds a rule, nobody ever removes one, and eighteen months later the system message is three thousand words holding at least two direct contradictions the model resolves by coin flip. The symptoms: behavior that changes when you reorder paragraphs, and rules that hold in testing but vanish under long context. The cure is subtraction with a safety net — cut instructions in batches and rerun your eval suite; most of the sink turns out to be superstition.