Your code calls json.loads() on a model reply and gets JSONDecodeError: Expecting value: line 1 column 1. Or it parses, and a field is a string where you wanted a number. This post sorts the causes by what you can safely do about each, gives a tested Python function that repairs only the unambiguous cases, and shows a retry loop that knows when a retry can’t help.
Choosing between JSON mode and json_schema is covered in JSON mode vs json_schema. This post is about what to do with the reply you actually got.
Why do LLMs return invalid JSON?
A model writes text one token at a time. Unless the server constrains decoding, nothing stops it adding a friendly sentence or running out of room. The common failures:
| What you see | Cause | Safe to repair in code? |
|---|---|---|
```json … ``` around the object |
Chat formatting habit | Yes: strip the fence |
| “Here is the JSON:” before, “Let me know…” after | Same habit | Yes: read the first value only |
<think>...</think> before the object |
Reasoning text in the content field on some servers | Yes: remove the block |
| Ends mid-string, brackets left open | Hit max_tokens |
No: resend with more room |
| Empty content | Reasoning used the whole budget | No: resend with more room |
{"a": 1,} |
Trailing comma; JavaScript allows it, RFC 8259 does not | Yes: one meaning |
{'a': True, 'b': None} |
Python syntax | Yes: ast.literal_eval reads it safely |
NaN, Infinity |
Not JSON; Python accepts them by default | No: reject |
| Valid JSON, wrong types or missing fields | Schema not enforced | No: re-ask with the error |
The dividing line is ambiguity. If there is exactly one thing the model could have meant, fix it in code and log that you did. If you would have to guess, reject the reply.
Why not repair truncated JSON?
Because the repaired version looks right and isn’t. We ran json_repair 0.64.9, a popular MIT-licensed library, on a reply cut off mid-word:
'{"name": "Omar", "city": "Jed' -> {'name': 'Omar', 'city': 'Jed'}
It returned valid JSON with the city “Jed”. Its README is open about this: it “automatically completes missing values in JSON fields with reasonable defaults”. That is useful for reading logs. For data you store or act on, it turns a visible failure into a silent one.
Check finish_reason before you parse. "length" means the reply hit max_tokens. Resending the same request with the same limit fails the same way, so double the limit or ask for less per call. Reasoning models make this more common: BroadNet’s API reference says DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, and the reasoning counts toward max_tokens. With a small limit, the answer can come back empty. Leave 1,000 tokens or more, or set reasoning_effort to low; the reasoning budget post has the numbers.
A repair function that only fixes the safe cases
This uses only the standard library. It returns the parsed value and a list of the repairs it made, so you can count them.
import ast, json, re
class Truncated(Exception): pass
class NotJSON(Exception): pass
FENCE = re.compile(r"```(?:json|JSON)?\s*\n(.*?)```", re.S)
THINK = re.compile(r"<think>.*?</think>", re.S)
def _reject_constant(name): # NaN, Infinity: Python accepts them, JSON does not
raise NotJSON(f"non-standard constant {name}")
_decoder = json.JSONDecoder(parse_constant=_reject_constant)
def _first_value(s):
"""Decode the first JSON object or array in s, ignoring prose before and after it."""
for i, ch in enumerate(s):
if ch in "{[":
try:
return _decoder.raw_decode(s, i)[0]
except json.JSONDecodeError:
continue # a stray brace in prose; try the next one
raise NotJSON("no JSON object or array found")
def _scan(s):
"""Drop commas before } or ] outside strings; report brackets left open."""
out, stack, in_str, esc = [], [], False, False
for i, ch in enumerate(s):
if in_str:
if esc:
esc = False
elif ch == "\\":
esc = True
elif ch == '"':
in_str = False
elif ch == '"':
in_str = True
elif ch in "{[":
stack.append(ch)
elif ch in "}]" and stack:
stack.pop()
elif ch == ",":
rest = s[i + 1:].lstrip()
if rest[:1] in ("}", "]"):
continue # trailing comma: skip it
out.append(ch)
return "".join(out), len(stack) + (1 if in_str else 0)
def _python_literal(s):
"""Output in Python syntax: single quotes, True/False/None, trailing commas."""
start = min((i for i in (s.find("{"), s.find("[")) if i >= 0), default=-1)
end = max(s.rfind("}"), s.rfind("]"))
if start < 0 or end <= start:
raise NotJSON("no braces")
value = ast.literal_eval(s[start:end + 1]) # literals only; never runs code
if not isinstance(value, (dict, list)):
raise NotJSON("not an object or array")
json.dumps(value, allow_nan=False) # must round-trip as strict JSON
return value
def parse_llm_json(content, finish_reason=None):
"""Return (value, repairs). Raises Truncated or NotJSON."""
if finish_reason == "length":
raise Truncated("hit max_tokens; raise the limit or shorten the output")
if not content or not content.strip():
raise NotJSON("empty content (reasoning may have used the whole budget)")
repairs = []
s = content.strip().lstrip("\ufeff")
if THINK.search(s):
s = THINK.sub("", s).strip(); repairs.append("removed <think> block")
m = FENCE.search(s)
if m:
s = m.group(1).strip(); repairs.append("removed code fence")
try:
value = _decoder.decode(s)
return value, repairs
except json.JSONDecodeError:
pass
try:
value = _first_value(s); repairs.append("ignored text around JSON")
return value, repairs
except NotJSON:
pass
fixed, open_count = _scan(s)
if open_count:
raise NotJSON(f"incomplete JSON, {open_count} bracket(s) or string left open: treat as truncated")
if fixed != s:
try:
value = _first_value(fixed); repairs.append("removed trailing commas")
return value, repairs
except NotJSON:
pass
try:
value = _python_literal(s); repairs.append("parsed Python-style literal")
return value, repairs
except (ValueError, SyntaxError, TypeError, MemoryError, RecursionError, NotJSON) as e:
raise NotJSON(f"unrepairable: {e}") from None
How each step works:
- Truncation and empty replies raise before any parsing, so they never become data.
- Fences and reasoning blocks are stripped with two regular expressions.
raw_decodeis the key tool. Python’s docs say it decodes a document “from a string that may have extraneous data at the end”, and returns where it stopped. Starting at the first{or[, it skips prose before the object and ignores prose after it. If a brace in prose fails to decode, it tries the next one.- The bracket scan walks the text tracking strings and escapes. Brackets left open mean the reply was cut off even if
finish_reasondidn’t say so; that raises. Commas directly before}or]outside strings are dropped. ast.literal_evalreads Python-style output with single quotes,TrueandNone. It evaluates literals only and never runs code. The result must then serialise as strict JSON.NaNandInfinityraise, throughparse_constant. Python’sjsonaccepts them by default; RFC 8259 says they “are not permitted”, and JavaScript’sJSON.parserejects them.
We ran it on 17 test strings: clean JSON, fenced JSON, Arabic keys and values in a fence, prose before and after, a {name} placeholder in prose before the object, a <think> block, trailing commas in objects and arrays, commas and brackets inside strings, escaped quotes, a Windows path ending in a backslash, Python literals, NaN, an empty reply, a reply cut off with and without finish_reason: "length", and two objects in a row. All behaved as the table above says. One test caught a real bug: an early version flipped its escape flag before checking for the closing quote, so "he said \"hi,]\"" lost its comma. Test your own repair code on escaped quotes.
Two limits to know. Given two objects in a row, it returns the first; if you want several items, ask for one JSON array. And single-quoted Python output that contains a double quote inside a string can be misread by the bracket scan and rejected. Rejecting is the safe failure.
How should you retry after repair fails?
Repair handles format. Validation handles content: right fields, right types, sane values. This loop validates with the jsonschema package and picks the retry by cause:
import json, os
from jsonschema import Draft202012Validator
from openai import OpenAI
from llm_json import parse_llm_json, Truncated, NotJSON
client = OpenAI(base_url=os.environ.get("BASE_URL", "https://inference.broadnet.ai/v1"),
api_key=os.environ["BROADNET_API_KEY"])
SCHEMA = {"type": "object", "additionalProperties": False,
"required": ["name", "city", "amount"],
"properties": {"name": {"type": "string", "minLength": 1},
"city": {"type": ["string", "null"]},
"amount": {"type": "number", "minimum": 0}}}
validator = Draft202012Validator(SCHEMA)
def extract(text, model="broadnet/qwen3.8-27b", max_tokens=1500, attempts=3, log=print):
messages = [{"role": "system", "content": "Extract name, city and amount. Reply with one JSON object only, no code fence. "
"Schema: " + json.dumps(SCHEMA)},
{"role": "user", "content": text}]
for attempt in range(1, attempts + 1):
r = client.chat.completions.create(model=model, messages=messages, max_tokens=max_tokens,
reasoning_effort="low", temperature=0)
c = r.choices[0]
try:
value, repairs = parse_llm_json(c.message.content, c.finish_reason)
errors = [f"{'/'.join(map(str, e.path)) or '(root)'}: {e.message}" for e in validator.iter_errors(value)]
if not errors:
log({"attempt": attempt, "repairs": repairs, "ok": True})
return value
problem = "The JSON does not match the schema: " + "; ".join(errors)
except Truncated:
log({"attempt": attempt, "ok": False, "why": "truncated"})
max_tokens *= 2 # same prompt, more room; a re-ask would be cut off too
continue
except NotJSON as e:
problem = f"That reply was not valid JSON ({e})."
log({"attempt": attempt, "ok": False, "why": problem})
messages += [{"role": "assistant", "content": c.message.content or ""},
{"role": "user", "content": problem + " Return the corrected JSON object only."}]
raise RuntimeError(f"no valid JSON after {attempts} attempts")
The rules in that loop:
- Truncated: resend the same messages with double the limit. Adding a “please fix” message would make the prompt longer and the output no shorter.
- Not JSON, or wrong shape: send the exact error back. “amount: ‘1200’ is not of type ‘number’” gives the model something specific to fix. A bare retry often repeats the mistake.
- Cap attempts at three. Every retry resends the whole conversation as input tokens. After three, log the input and route it elsewhere.
- Log the repairs list. A rising count of “removed code fence” after a prompt change tells you the prompt got worse, even though nothing failed.
We ran this loop against a local mock server that played back scripted replies. A fenced reply with a trailing comma and "amount": "1200" as a string was repaired, failed validation, and was fixed on the second call with the error sent back. A reply cut off with finish_reason: "length" was resent with max_tokens raised from 1,500 to 3,000 and succeeded. The mock recorded the request each time, confirming the message count and limit the loop sent. A real model can fail differently, so run it on 100 of your own inputs before trusting the numbers.
Does structured output make this unnecessary?
It removes most of it. A server that constrains decoding to your schema cannot emit a fence, a trailing comma or a missing field. vLLM, a common open-model server, does this with the xgrammar or guidance backends. Two failures remain even then: truncation at max_tokens, and refusals. OpenAI notes that JSON mode output is valid “except for in some edge cases” and that your application “must detect and handle” output that is not a complete object. So keep the finish_reason check and the validator, even with json_schema.
What each approach prevents:
| Approach | Fences and prose | Syntax errors | Wrong shape | Truncation |
|---|---|---|---|---|
| Prompt only (“reply with JSON”) | Reduced | Reduced | Reduced | No |
JSON mode (json_object) |
Yes | Yes | No | No |
Structured output (json_schema, enforced) |
Yes | Yes | Yes | No |
With JSON mode, OpenAI also says you “must always instruct the model to produce JSON” in a message; otherwise it may emit whitespace until it hits the token limit. Other OpenAI-compatible servers vary, so test the mode on the model you ship. The test is in the structured output post.
To check a reply by hand, paste it into the LLM JSON output validator, or build a strict schema from a sample object with the JSON schema generator.
Where BroadNet Inference fits
BroadRouter is BroadNet’s site. BroadNet’s API reference lists structured output for GLM 5.3 and Qwen3.8 27B only, and does not name the request parameter, so test json_schema on the model before relying on it. On DeepSeek V4.1 Flash and Gemma 4 VL 26B, use prompt-only JSON with the repair function and validator above. Three of the four models reason before answering, so budget max_tokens for it. For bulk extraction, the batch runner post shows how to log finish_reason per item. Prompts and replies are not retained after each request.
