A vision model can read an invoice, but it can also misread a digit, skip a line or invent a field that isn’t there. The fix is not a better prompt alone. It is a review step designed in from day one: rules that decide which documents a person sees, a queue that tells them why, and a log that lets you prove the error rate.
This post covers that step. The extraction itself (image preparation, the JSON prompt, schema validation and retries) is in extracting data from documents with a vision LLM, and Arabic-specific reading is in reading Arabic invoices with vision. Here we assume you already get JSON back and need to decide what to trust.
What should decide whether a person reviews a document?
Rules you can test in code, not a feeling about the model. Start with these, in order of value:
| Rule | Example reason code | Why it matters |
|---|---|---|
| Required field missing | missing:total |
A null is honest; a guessed value is worse |
| Sums disagree | arithmetic:subtotal+tax!=total |
One misread digit almost always breaks a sum |
| Line items don’t add up | arithmetic:lines!=subtotal |
Catches skipped or duplicated lines |
| Date won’t parse or is out of range | date:unparsed |
Catches day/month swaps and OCR noise |
| Unknown vendor tax ID | vendor:unknown |
New suppliers and fraud both start here |
| Unexpected currency | currency:unexpected |
A misread symbol changes the amount |
| Model says it is unsure | model:not_confident |
A hint, never the only gate |
Two design choices make the queue usable. First, every rule produces a short, stable reason code, so a reviewer opens the document knowing what to check, and you can count which rule fires most. Second, a document either passes every rule or goes to a person. Avoid weighted scores that nobody can explain to an auditor.
Should you trust the model’s confidence?
Treat it as a weak signal. You can ask the model to rate each field as high, medium or low confidence, and it will. But Xiong et al. (ICLR 2024) benchmarked ways to get LLMs to state confidence and found that models “tend to be overconfident” when they say how sure they are; better prompting and sampling helped, and no method consistently beat the others. Token log-probabilities are another option on some APIs. BroadNet’s API reference does not document logprobs, so don’t design around them there.
The practical rule: anything below “high” from the model, or no rating at all, sends the document to review, but a “high” never skips the code checks. After a few hundred reviewed documents, check whether “high” fields really had fewer corrections. If they didn’t, drop the signal.
How do you route documents to a queue?
This router runs after extraction and validation. It normalises Arabic-Indic digits, applies the rules above and decides one of three queues: review, audit_sample or auto. It is plain Python with no API call, so you can unit-test it.
import hashlib, json, random, re, time
from datetime import date, datetime
from decimal import Decimal, InvalidOperation
REQUIRED = ["vendor_name", "invoice_number", "invoice_date", "currency", "subtotal", "tax", "total"]
CURRENCIES = {"SAR", "AED", "USD", "EUR"}
KNOWN_VENDORS = {"300000000000003": "Example Trading Co"} # your supplier master: tax ID -> name
AUTO_SAMPLE_RATE = 0.05 # share of clean documents a person still checks
DIGITS = str.maketrans("٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹٫٬", "01234567890123456789.,")
def money(v):
"""'١٬٢٥٠٫٥٠' or '1,250.50' -> Decimal('1250.50'); None if it is not a number."""
if v is None:
return None
s = str(v).translate(DIGITS).replace(",", "").strip()
try:
return Decimal(s)
except InvalidOperation:
return None
def check(doc: dict, today: date) -> list[str]:
"""Return the reasons this document needs a person. Empty list = may auto-approve."""
reasons = [f"missing:{f}" for f in REQUIRED if doc.get(f) in (None, "")]
sub, tax, tot = (money(doc.get(k)) for k in ("subtotal", "tax", "total"))
if None not in (sub, tax, tot) and abs(sub + tax - tot) > Decimal("0.01"):
reasons.append("arithmetic:subtotal+tax!=total")
lines = [money(l.get("amount")) for l in doc.get("lines", [])]
if lines and None not in lines and sub is not None and abs(sum(lines) - sub) > Decimal("0.01"):
reasons.append("arithmetic:lines!=subtotal")
try:
d = datetime.strptime(str(doc.get("invoice_date", "")).translate(DIGITS), "%Y-%m-%d").date()
if d > today or (today - d).days > 400:
reasons.append("date:out_of_range")
except ValueError:
reasons.append("date:unparsed")
if doc.get("currency") and doc["currency"] not in CURRENCIES:
reasons.append("currency:unexpected")
if doc.get("vendor_tax_id") not in KNOWN_VENDORS:
reasons.append("vendor:unknown")
if doc.get("model_confidence", "low") != "high": # a hint from the prompt, never the only gate
reasons.append("model:not_confident")
return reasons
def route(doc_id: str, doc: dict, today: date, rng=random.random) -> dict:
reasons = check(doc, today)
if reasons:
queue = "review"
elif rng() < AUTO_SAMPLE_RATE:
queue, reasons = "audit_sample", ["sample:random"]
else:
queue = "auto"
return {"doc": hashlib.sha256(doc_id.encode()).hexdigest()[:16], "queue": queue,
"reasons": reasons, "ts": int(time.time())}
We tested it on three made-up Saudi invoices (the vendor and tax ID are fictional). A clean one, with the date and subtotal written in Arabic-Indic digits (٢٠٢٦-١١-٠٢ and ١٬٠٠٠٫٠٠), went to auto. The same invoice with the total misread as 1,105.00 went to review with arithmetic:subtotal+tax!=total. One with an unknown tax ID, a month of 13 and a “medium” confidence went to review with three reasons: date:unparsed, vendor:unknown and model:not_confident. Forcing the random draw below 0.05 sent the clean invoice to audit_sample.
Two limits to know. The money() helper assumes a dot or Arabic decimal separator; a European “1.150,00” needs its own parser keyed on the vendor’s country. And the date rule assumes the model returned ISO dates. Hijri dates need conversion first, as in Arabic named entity extraction.
What changes for Arabic and English invoices?
Bilingual invoices are common in the Gulf: Arabic and English labels side by side, amounts in Western or Arabic-Indic digits, and the vendor name in two scripts. A few rules help:
- Normalise digits before any check. Map ٠–٩ and the Persian ۰–۹ to 0–9, and the Arabic decimal and thousands separators (٫ and ٬) to their ASCII forms. Then compare.
- Match vendors on the tax ID, not the name. The Arabic and English names rarely match your master record character for character. The registration number does.
- Keep both names for the reviewer. Show the extracted Arabic and English vendor names next to the master record, with the Arabic in a right-to-left, isolated block. The multilingual product design post covers bidi display.
- Check whether a structured invoice exists. Saudi Arabia’s ZATCA runs electronic invoicing (Fatoorah) in two phases, from 4 December 2021 and 1 January 2023, built around invoices in a structured electronic format. If a supplier can send you that, parse it instead of reading a picture of it, and keep vision for paper, photos and the long tail.
Arabic OCR with a vision LLM covers the reading side: diacritics, mixed scripts and layout.
What should the reviewer see?
A good review screen answers one question per reason code. For arithmetic, show the line items, subtotal, tax and total side by side with the computed sum. For vendor:unknown, show the extracted tax ID and the closest master records. For date:unparsed, show the raw date text and the crop of the image it came from.
Show the image next to the fields, not in another tab. Let the reviewer fix a field, approve, or reject the document as unreadable. Record which action they took. “Unreadable” is useful data: if 10% of a supplier’s documents are unreadable, ask that supplier for PDFs, not your model for a miracle.
Why sample documents that passed every rule?
Because rules only catch the errors you thought of. If a model misreads a line item, the subtotal and the total by the same amount (a smudged 7 read as 1 in all three places), every sum still agrees and no rule fires. The only way to find those errors is to look at documents nobody flagged.
Sample them at random, not by choice, and track how many needed a correction. A small sample gives a wide range. With the 95% Wilson interval, which the NIST handbook describes for proportions (citing recommendations by Agresti and Coull, and by Brown, Cai and DasGupta), our metrics code below gives:
| Audited documents with zero errors found | True error rate could still be up to |
|---|---|
| 50 | about 7.1% |
| 300 | about 1.3% |
So “we audited 50 and found nothing” does not mean the error rate is below 1%. Raise the sample after any change to the prompt, the model or the document mix, and lower it slowly.
How do you keep an audit trail?
Record corrections in your own system. Each document needs a row with:
- a SHA-256 hash of the source file (and where the file itself is stored, under your retention rules);
- the model ID, prompt version and the raw JSON the model returned;
- the provider’s request ID (BroadNet returns it in the
X-Request-IDheader) for support questions; - the rule results and the queue;
- the reviewer, the time, the action and the before and after value of every corrected field.
This matters more with a zero-retention API. BroadNet’s privacy statement says its gateway keeps no prompts, images, responses or transcripts, and keeps only content-free account, billing, traffic and security records. That is good for sensitive invoices, but it means nobody at the provider can show you what the model saw or said last month. Your log is the only record. Keep it free of anything you don’t need; LLM API usage logging covers what to log and what to redact.
How do you measure the review rate?
Run this weekly over the routing log, with corrections joined in:
import math
from collections import Counter
def wilson(errors: int, n: int, z: float = 1.96) -> tuple[float, float]:
"""95% Wilson score interval for an error rate seen in a sample."""
if n == 0:
return (0.0, 1.0)
p = errors / n
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return (max(0.0, centre - half), min(1.0, centre + half))
def report(log: list[dict]) -> dict:
queues = Counter(r["queue"] for r in log)
reasons = Counter(x.split(":")[0] for r in log if r["queue"] == "review" for x in r["reasons"])
audited = [r for r in log if r["queue"] == "audit_sample"]
wrong = sum(1 for r in audited if r.get("corrected_fields"))
lo, hi = wilson(wrong, len(audited))
return {"documents": len(log),
"review_rate": round(queues["review"] / len(log), 3),
"top_reasons": reasons.most_common(3),
"audited": len(audited), "audited_with_errors": wrong,
"auto_error_rate_95ci": (round(lo, 4), round(hi, 4))}
On a synthetic week of 1,000 documents (760 auto, 190 review, 50 audited of which 2 needed a correction), it reported a review rate of 0.19, vendor as the top reason with 120, and an audited error rate between 1.1% and 13.5%. That last range is the honest answer to “how accurate is the automation?” at this sample size.
Read the three numbers together:
- Review rate falls, audited errors stay flat: a rule was too strict, or the model improved. Good.
- Review rate falls, audited errors rise: a rule is too loose. Tighten it before the errors reach your ledger.
- One reason dominates: fix the cause. If
vendor:unknownleads, update the supplier master; ifarithmeticleads, look at image quality or try another model on the same documents with the method in evaluating an LLM before switching.
Where BroadNet Inference fits
BroadRouter is BroadNet’s own site. BroadNet Inference offers three chat models that take images, according to its pricing feed: Qwen3.8 27B ($0.175 input / $2.17 output per 1M tokens, with structured output documented), Gemma 4 VL 26B ($0.10 / $0.22, usable on the $5 starting credit) and DeepSeek V4.1 Flash ($0.175 / $0.52). On pay-as-you-go, images count as input tokens; the image token calculator helps estimate them. Inference content is not retained; necessary account, billing and security metadata is retained as described in BroadNet’s privacy statement. BroadNet does not provide a review queue or audit log, so the routing and records above are yours to build and keep.
