Computer Vision · entry 04/05

OCR & document AI

Reading text in images is nearly solved; reading documents — tables, forms, totals, the fields money rides on — is a systems problem where the last percent is the whole job.

From characters to layouts

Classic OCR — find text lines, recognize characters, output strings — is mature technology: on clean print it exceeds 99% character accuracy, and open engines give it away. But a document is not a string. An invoice is a layout: a vendor block, a line-item table, a total that must equal the sum, a due date living in a corner. Document AI is the discipline of recovering that structure — reading order, tables, key–value pairs — and it's where the useful systems distinguish themselves, because business data lives in structure, not prose.

The modern pipeline

Today's strongest approach reads the page directly with a vision-language model: render each page to an image, prompt with the extraction schema, get structured output back.

Extract from this invoice image:
{ "vendor": str, "invoice_no": str, "date": "YYYY-MM-DD",
  "line_items": [{"desc": str, "qty": num, "unit_price": num}],
  "total": num }
Rules: transcribe exactly what is printed. If a field is not
visible, use null. Do not compute missing values.

The "null, don't compute" rule is load-bearing: without it, models helpfully derive totals that were actually illegible. Wrap the call in validation — totals must sum, dates must parse, IDs must match their checksums — and route validation failures to a human. For high volumes of one stable form, specialized layout models are cheaper per page; for the long tail of formats, the VLM's flexibility wins.

Measure fields, not characters

Vendors quote character accuracy because it flatters. At 99% character accuracy, a ten-digit amount field is wrong about one time in ten — and unlike prose, numbers offer no redundancy: a misread "7" doesn't look wrong in context. The honest metric is field-level: what fraction of totals, dates, account numbers come out exactly right, measured on your own documents, scans and phone photos included. Field accuracy is always startlingly lower than character accuracy; budget from the former.

Failure mode

Confident misreads of exactly the fields that matter. A smudged 7 becomes a 1 with full confidence; a crossed-out line is transcribed as current; a second-page total overrides the first. Systems fail silently and plausibly — the output parses, the schema validates, the money is wrong. Production document AI is therefore a triage system: arithmetic and checksum validation first, model-reported uncertainty second, sampled human review always, and the sampling rate set by what one wrong field costs you.