Your code needs data, and models write text. Structured output closes that gap, but “the model returns JSON” means three different things depending on how you ask. This post compares the options on OpenAI-compatible open models, shows how to request each, and gives a validate-and-repair loop you can drop into production.
What is the difference between JSON mode and json_schema?
| Method | Request | What you’re promised |
|---|---|---|
| Prompt only | “Reply in JSON with fields…” | Nothing; usually JSON, sometimes with extra text |
| JSON mode | response_format: {"type": "json_object"} |
Valid JSON, any shape |
| Structured output | response_format: {"type": "json_schema", ...} |
JSON matching your schema, where enforced |
OpenAI’s Structured Outputs guide puts it directly: “While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence.” It recommends structured output over JSON mode whenever possible.
On open-model providers, the guarantee depends on how the server implements it. Many use constrained decoding, which blocks tokens that would break the schema. Some accept the field but don’t enforce it. That’s why the code below validates regardless.
How do you request json_schema?
import os
from openai import OpenAI
from pydantic import BaseModel, Field
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
class Invoice(BaseModel):
vendor: str
currency: str = Field(pattern="^[A-Z]{3}$")
total: float
line_items: list[float]
schema = Invoice.model_json_schema()
schema["additionalProperties"] = False
response_format = {"type": "json_schema",
"json_schema": {"name": "invoice", "strict": True, "schema": schema}}
Pydantic generates the JSON Schema from your class, so the schema and the validator never drift apart. If you don’t use Pydantic, the JSON schema generator builds a strict-mode schema from a sample object. For strict mode, OpenAI requires additionalProperties: false and every property listed as required.
Which mode does your provider support?
Run this test once per model before you build on it:
- Send
json_schemawith a prompt that invites an extra field (“also add a short comment”). - If the request errors, the provider doesn’t accept
json_schema. Tryjson_object. - If the reply contains the extra field, the schema isn’t enforced. Treat it as JSON mode.
- If
json_objectalso errors, use prompt-only with validation.
On BroadNet Inference, the docs list structured output for GLM 5.3 and Qwen3.8 27B (compared side by side), but don’t name the request parameter. Run the test above on the model you ship.
How do you validate and repair bad output?
import json
from pydantic import ValidationError
def extract(text: str, model="broadnet/glm-5.3", attempts=3) -> Invoice:
messages = [
{"role": "system", "content": "Extract the invoice. Reply with JSON only."},
{"role": "user", "content": text},
]
for attempt in range(attempts):
r = client.chat.completions.create(model=model, messages=messages,
response_format=response_format,
max_tokens=4000)
choice = r.choices[0]
if choice.finish_reason == "length":
raise RuntimeError("Cut off: raise max_tokens") # retrying won't help
if getattr(choice.message, "refusal", None):
raise RuntimeError(f"Refused: {choice.message.refusal}")
raw = choice.message.content or ""
try:
inv = Invoice.model_validate_json(raw)
if abs(sum(inv.line_items) - inv.total) > 0.01:
raise ValueError("total must equal the sum of line_items")
return inv
except (ValidationError, ValueError) as e:
messages += [
{"role": "assistant", "content": raw},
{"role": "user", "content": f"That JSON is invalid: {e}. Return corrected JSON only."},
]
raise RuntimeError(f"No valid JSON after {attempts} attempts")
What makes this loop work:
- Check
finish_reasonfirst. A reply that hitmax_tokensis truncated JSON. Retrying with the same limit fails the same way. Both models above reason before answering, and reasoning counts toward the limit, so leave room. - Handle refusals separately. OpenAI’s guide notes a refusal “does not necessarily follow the schema you have supplied”.
- Send the exact error back. A blind retry often returns the same mistake. “line_items: field required” gives the model something to fix.
- Check business rules in code. A schema can’t say a total must equal its line items. The check above can.
- Cap attempts. Every attempt resends the whole prompt as input tokens.
What schema features cause trouble?
Keep schemas boring:
- Flat beats nested. Two levels is plenty for most extraction.
- Enums over free strings for categories.
- No open-ended objects. Free-form
dictfields defeat the point. - Short field names with descriptions. Descriptions in the schema guide the model.
- Optional fields as nullable. In strict mode, list every field as required and allow
null.
OpenAI’s guide notes some JSON Schema features are unsupported in strict mode. Open-model servers have their own limits. If a schema is rejected, simplify it before you switch modes.
How do you monitor structured output in production?
A loop that repairs silently can hide a problem that costs you tokens every day. Log three numbers per call site:
| Metric | What it tells you |
|---|---|
| First-attempt success rate | How well the model and schema fit; a drop often follows a prompt or model change |
| Average attempts per success | The token cost of repairs |
| Final failures | Inputs your pipeline can’t handle; review them by hand |
If first-attempt success falls below what you’re used to, simplify the schema or try another model before you add more retries. Keep a sample of failed raw replies, so you can test fixes against real cases.
When should you use a tool instead?
Use structured output for the final answer: an extraction, a classification, a plan. Use tool calling when the model needs your code to do something and continue. Many agents use both: tools during the loop, a schema for the last step. For an image-to-JSON example, see reading Arabic invoices with a vision model.
Where BroadNet Inference fits
BroadNet Inference lists structured output for GLM 5.3 and Qwen3.8 27B, behind the standard chat-completions format, so the code above runs with the OpenAI Python SDK unchanged. Qwen3.8 27B also takes images, which suits document extraction. BroadNet’s docs don’t state structured output for DeepSeek V4.1 Flash or Gemma 4 VL 26B, so use the prompt-only path with validation there. Compare per-token prices on the pricing page.
