---
title: "Extract data from invoices to JSON with a vision LLM"
description: "A production pipeline for invoices, receipts and forms: image prep, a JSON schema prompt, Pydantic validation, arithmetic checks, retries and a human review queue."
url: https://broadrouter.com/blog/extract-data-from-documents-with-vision-llm/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Extract invoice, receipt and form data to JSON with a vision LLM

By the BroadNet Inference team · 4 min read · Published 2026-11-14

Render each page to an image, send it to a vision model with a prompt that names every field and its format, and ask for JSON only. Then validate in code: parse it, check its schema, check that line items add up to the total, and retry once with the error. Route repeat failures to a person.

## Key takeaways

-   The model reads; your code decides. Never write extracted data to a ledger without schema and arithmetic checks.
-   Name every field, its type and its format in the prompt, and say what to return when a field is missing (null, never a guess).
-   Arithmetic checks catch most misreads: line totals must sum to the subtotal, and subtotal plus tax must equal the total.
-   Retry once with the validation error in the prompt; route repeat failures to a human queue.
-   On pay-as-you-go, images bill as input tokens and larger images use more, so resize before sending.

**On this page**

1.  What does the pipeline look like?
2.  How do you prepare document images?
3.  How should you write the extraction prompt?
4.  How do you get structured output and validate it?
5.  Which checks catch the most errors?
6.  How do you measure accuracy before going live?
7.  Which vision models fit on BroadNet Inference?
8.  Where BroadNet Inference fits

_[image: A paper invoice headed "فاتورة" (invoice) with tabs labelled Vendor, Date and Total, and the text "Turn documents into useful data."]_

Illustrative example.

Invoices, receipts and forms all have the same problem: the data is in a picture, and you need it in a database. Template-based OCR breaks on every new vendor layout. A [vision-language model](https://broadrouter.com/glossary/vision-language-model/) reads the page and returns fields directly. The catch is that it can be confidently wrong, so the work is in the pipeline around it. This post covers the general pipeline for any language. For Arabic documents, see [reading Arabic invoices with vision](https://broadrouter.com/guides/read-arabic-invoices-with-vision/) for setup and [Arabic OCR vs vision LLMs](https://broadrouter.com/blog/arabic-ocr-vision-llm/) for when classic OCR is the better tool.

## What does the pipeline look like?

1.  **Prepare** each page: render PDFs to images, fix rotation, resize.
2.  **Extract**: one request per document with a field-by-field prompt, JSON only.
3.  **Validate**: parse, schema-check, arithmetic-check.
4.  **Retry** once with the error message if validation fails.
5.  **Review**: anything still failing goes to a person, with the image and the draft.
6.  **Measure** field accuracy on a labelled sample, and re-run it whenever you change model or prompt.

## How do you prepare document images?

Render PDFs to images with PyMuPDF and keep the smallest text sharp without sending more pixels than needed:

```python
# pip install pymupdf pillow
import base64, io
import pymupdf  # older code imports it as fitz
from PIL import Image

def pdf_pages_as_data_urls(path, long_edge=1800):
    urls = []
    for page in pymupdf.open(path):
        pix = page.get_pixmap(dpi=200)
        img = Image.open(io.BytesIO(pix.tobytes("png"))).convert("RGB")
        img.thumbnail((long_edge, long_edge))
        buf = io.BytesIO()
        img.save(buf, format="JPEG", quality=90)
        urls.append("data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode())
    return urls
```

On BroadNet Inference, images go in the message as `data:` URLs with an accurate MIME type, and on [pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/) they bill as input tokens; larger images use more. Measure tokens per page once with the `usage` block and size your batches from that; [counting tokens before sending](https://broadrouter.com/blog/count-tokens-before-sending/) shows how, and [vision LLM cost per image](https://broadrouter.com/blog/cheapest-vision-llm-api/) turns tokens into cost per page. To estimate tokens per page before you send anything, drop a sample scan into the [image token calculator](https://broadrouter.com/tools/image-token-calculator/).

## How should you write the extraction prompt?

Be exact about keys, types, formats and missing values. If you also want a JSON Schema for validation or for providers that enforce one, the free [JSON schema generator](https://broadrouter.com/tools/json-schema-generator/) builds it from a sample invoice:

```python
PROMPT = """Extract this invoice. Reply with one JSON object only, no prose.
Keys:
- vendor_name: string, as printed
- vendor_tax_id: string or null
- invoice_number: string
- invoice_date: string, YYYY-MM-DD
- currency: ISO 4217 code, e.g. USD, EUR, SAR
- line_items: list of {description: string, quantity: number, unit_price: number, line_total: number}
- subtotal: number
- tax: number or null
- total: number
Rules: numbers without currency symbols or thousands separators. If a field is not on the page, use null. Never guess."""
```

“Never guess” plus an explicit `null` matters most. Without it, models fill gaps with plausible values, which is worse than a blank because it passes a naive check.

## How do you get structured output and validate it?

There are two levels of [structured output](https://broadrouter.com/glossary/structured-output/). JSON mode guarantees valid JSON. Schema-constrained output also guarantees the shape. OpenAI’s guide puts it plainly: both produce valid JSON, but only Structured Outputs ensure schema adherence. Support for either varies by provider and model, so validate in code whatever the API promises:

```python
import json, os
from decimal import Decimal
from pydantic import BaseModel, ValidationError, model_validator
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

class Line(BaseModel):
    description: str
    quantity: Decimal
    unit_price: Decimal
    line_total: Decimal

class Invoice(BaseModel):
    vendor_name: str
    vendor_tax_id: str | None = None
    invoice_number: str
    invoice_date: str
    currency: str
    line_items: list[Line]
    subtotal: Decimal
    tax: Decimal | None = None
    total: Decimal

    @model_validator(mode="after")
    def sums(self):
        tol = Decimal("0.02")
        if abs(sum(l.line_total for l in self.line_items) - self.subtotal) > tol:
            raise ValueError("line_totals do not sum to subtotal")
        if abs(self.subtotal + (self.tax or 0) - self.total) > tol:
            raise ValueError("subtotal + tax != total")
        return self

def extract(image_urls, model="broadnet/qwen3.8-27b"):
    content = [{"type": "text", "text": PROMPT}] + [
        {"type": "image_url", "image_url": {"url": u}} for u in image_urls]
    messages = [{"role": "user", "content": content}]
    for attempt in range(2):
        r = client.chat.completions.create(model=model, messages=messages, max_tokens=4000, temperature=0)
        raw = (r.choices[0].message.content or "").strip().removeprefix("```json").removesuffix("```")
        try:
            return Invoice.model_validate(json.loads(raw)), r.usage
        except (json.JSONDecodeError, ValidationError, ValueError) as e:
            messages += [{"role": "assistant", "content": raw},
                         {"role": "user", "content": f"That failed validation: {e}. Re-read the image and reply with corrected JSON only."}]
    return None, r.usage  # send to human review
```

Use `Decimal`, not `float`, for money. Keep `max_tokens` generous: Qwen3.8 27B reasons before answering, and reasoning counts toward the limit; a small cap returns an empty string. See [why reasoning models return empty replies](https://broadrouter.com/guides/reasoning-tokens-empty-response/).

## Which checks catch the most errors?

| Check | Catches | Cost |
| --- | --- | --- |
| JSON parses | Prose around the JSON, truncation | Free |
| Schema validates | Missing keys, wrong types | Free |
| Lines sum to subtotal | Misread digits, skipped lines | Free |
| Subtotal + tax = total | Misread total or tax | Free |
| Date parses and is not in the future | Day/month swaps, OCR noise | Free |
| Vendor tax ID matches your supplier list | Wrong vendor, fraud | One lookup |
| Duplicate invoice number per vendor | Double payment | One query |

Arithmetic checks are the best value: a single misread digit almost always breaks a sum. Receipts often lack line-level detail, so check what is present and accept fewer checks for them.

## How do you measure accuracy before going live?

Label 50 to 100 real documents by hand, run the pipeline, and score **per field**, not per document. Totals and dates deserve a stricter bar than descriptions. Track three numbers: field accuracy, the share of documents that pass validation without retry, and the share sent to review. How to design that review step, including random audits of documents that passed, is in [making human review part of your document AI workflow](https://broadrouter.com/blog/document-review-queues/). When you try another model, run the same set; [evaluating an LLM before switching](https://broadrouter.com/blog/evaluate-llm-before-switching/) covers the method.

## Which vision models fit on BroadNet Inference?

| Model | Images | Context | Notes |
| --- | --- | --- | --- |
| [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) | Yes | 262,144 | Tools, structured output and reasoning listed; tested in Arabic by BroadNet |
| [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/) | Yes | 32,768 | Lowest BroadNet price; runs on the free credit and the hourly key |
| [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) | Yes | 1,048,576 | Images included on the unlimited plans |

Prices per 1M tokens are on the [pricing page](https://broadrouter.com/pricing/); multiply your measured tokens per page by monthly volume in the [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/).

## Where BroadNet Inference fits

BroadNet Inference takes images on three chat models through the standard OpenAI message format, and it does not retain prompts, images or answers after the request, which matters for invoices full of names, tax IDs and bank details ([zero data retention](https://broadrouter.com/zero-data-retention/)). Its docs list structured output for Qwen3.8 27B but do not document schema parameters, so rely on the validation layer above rather than on the API to enforce your schema. BroadNet does not publish an image size limit or a dedicated document-AI product, and its docs describe image input, not PDF upload, so render pages to images first. For high-volume runs on a fixed budget, an [unlimited DeepSeek V4.1 Flash plan](https://broadrouter.com/guides/unlimited-llm-api-sessions/) includes images.

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Structured output](https://broadrouter.com/glossary/structured-output/)

Structured output is an LLM API feature that constrains a model's reply to a format the developer specifies, usually a JSON Schema, so code can parse it reliably. It is stricter than JSON mode, which guarantees valid JSON but not a particular shape.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[JSON Schema](https://broadrouter.com/glossary/json-schema/)

JSON Schema is a vocabulary, written in JSON, for describing and validating the structure of JSON data: types, required fields, allowed values and nesting. The current version is 2020-12. LLM APIs use it to define tool parameters and the exact shape a structured output must follow.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Vision-language model](https://broadrouter.com/glossary/vision-language-model/)

A vision-language model (VLM) is a model that takes images and text together as input and produces text, so it can describe pictures, answer questions about them and read documents, charts or screenshots. VLMs are also called image-text-to-text or multimodal models.

[Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/)

Input tokens are the tokens a language model reads in a request, including the system prompt, conversation history, tool definitions and images. Output tokens are the tokens it generates in reply, including any reasoning. LLM APIs usually price them separately, and output usually costs more.

[JSON mode](https://broadrouter.com/glossary/json-mode/)

JSON mode is an LLM API setting, response\_format {"type": "json\_object"} in OpenAI-style APIs, that makes the model's reply valid, parseable JSON. Unlike structured outputs with a JSON Schema, it does not guarantee which keys, types or values the object contains.

## Questions

### Can a vision LLM replace OCR for invoices?

For many layouts, yes. A vision model reads the image and maps text to fields in one step, without templates per vendor. It still misreads digits sometimes, so keep validation and a review queue.

### How do I get reliable JSON from a vision model?

List the exact keys, types and formats in the prompt, ask for JSON only, and parse it in code. If the model supports schema-constrained output, use it. Either way, validate with a schema library and retry once with the error.

### How do I extract data from a multi-page PDF invoice?

Render each page to an image, then send all pages of one invoice in a single request if they fit the model's context, so totals on the last page can be matched to lines on the first.

### What resolution should invoice images be?

High enough that the smallest digits are sharp, and no higher. Larger images use more input tokens. A long edge of about 1,500 to 2,000 pixels is a sensible starting point for an A4 page; test on your own documents.

### How do I measure extraction accuracy?

Label 50 to 100 real documents by hand and score field by field. Track totals and dates separately from free-text fields like vendor names, because their error costs differ.

## Keep reading

-   [Arabic OCR vs vision LLMs: reading Arabic documents and invoices](https://broadrouter.com/blog/arabic-ocr-vision-llm/)
-   [Arabic OCR API comparison: Google, Azure, Textract, Mistral, LLMs](https://broadrouter.com/blog/arabic-ocr-api-comparison/)
-   [Cheapest vision LLM API: cost per image and per page, worked out](https://broadrouter.com/blog/cheapest-vision-llm-api/)
-   [JSON mode vs json\_schema: structured output that holds up](https://broadrouter.com/blog/structured-output-json-schema/)
-   [How to evaluate an LLM before switching providers](https://broadrouter.com/blog/evaluate-llm-before-switching/)
-   [How to count tokens before you send an LLM request](https://broadrouter.com/blog/count-tokens-before-sending/)
-   [Make human review part of your document AI workflow](https://broadrouter.com/blog/document-review-queues/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: Read Arabic invoices and receipts with a vision model](https://broadrouter.com/guides/read-arabic-invoices-with-vision/)
-   [Guide: Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-extract-data-from-documents-with-vision-llm&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference: chat with an image, MIME types, images on plans (fetched 2026-11-14)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-extract-data-from-documents-with-vision-llm&utm_content=sources#requests)
-   [BroadNet Inference machine-readable prices, models and modalities, pricing.json (fetched 2026-11-14)](https://inference.broadnet.ai/pricing.json?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-extract-data-from-documents-with-vision-llm&utm_content=sources)
-   [OpenAI Structured Outputs guide, JSON mode vs schema adherence (fetched 2026-11-14)](https://developers.openai.com/api/docs/guides/structured-outputs)
-   [Pydantic documentation, models and validation (fetched 2026-11-14)](https://docs.pydantic.dev/latest/concepts/models/)
-   [PyMuPDF documentation, rendering pages to images (fetched 2026-11-14)](https://pymupdf.readthedocs.io/en/latest/recipes-images.html)
