Prompt Engineering · entry 03/04

Structured output

Three mechanisms turn a text generator into a JSON API — and the schema itself is the most underrated prompt you will write.

Prose is for people

The moment model output feeds a program instead of a person, format stops being a preference and becomes an interface contract. "Return JSON" alone gets you JSON most of the time — wrapped in markdown fences, or prefixed with a friendly "Here is the JSON you requested," or carrying one trailing comma, each of which is a parser crash at 2 a.m. Structured output is the discipline of closing that gap to zero.

Three mechanisms, in rising order of strength

First, prompting: show the exact schema, demand nothing but JSON, give one worked example. Cheap and portable, still probabilistic. Second, function calling: define a tool whose parameters are your schema, and the API hands back arguments as parsed JSON — a perfectly good output channel even when no real function exists on your side. Third, constrained decoding — JSON modes and strict structured-output settings — where the runtime masks token choices so that nothing outside the grammar can be sampled at all. Syntactic validity becomes guaranteed; correctness, as ever, does not.

The schema is a prompt

Models read schemas the way they read everything else, so write field names and descriptions as micro-instructions: refund_eligible with a description stating the policy beats flag2 forever. Prefer enums to free strings wherever the set is closed. Keep schemas small — five fields the model fills thoughtfully beat forty it pads with nulls — and order the fields so the tokens that think come before the tokens that decide:

{
  "type": "object",
  "properties": {
    "reasoning": {"type": "string",
      "description": "One sentence citing the policy rule applied."},
    "refund_eligible": {"type": "boolean"},
    "amount_usd": {"type": "number", "minimum": 0}
  },
  "required": ["reasoning", "refund_eligible", "amount_usd"]
}

Validate, then retry

Whatever the mechanism, validate every response against the schema at the application boundary, and on failure retry once with the validator's error appended — "amount_usd was a string; return a number" fixes itself on the second pass a large fraction of the time. Cap the loop at one or two attempts and fail loudly after. Log every rejection: a rising invalid-output rate is the cheapest regression signal production will ever hand you.

Failure mode

Mistaking valid for true. Constrained decoding satisfies the schema no matter what — so a model with nothing sensible to say fills your required fields with confident garbage instead of the hedge or refusal it would have produced in prose. Structure removes exactly one class of error, the parse error, and quietly converts the rest into well-typed hallucinations. Build honesty into the schema itself: a nullable answer, a confidence field, an explicit "insufficient_information" enum value.