BroadRouter
Get started

Tool calling with open models: schemas, parallel calls, retries

By the BroadNet Inference team · 4 min read · Published · Updated

Open models on OpenAI-compatible APIs use the same tools and tool_calls format as OpenAI. Keep schemas small and flat, parse arguments as JSON strings, validate them before running anything, and return errors to the model as tool results so it can retry. Cap the steps. On BroadNet, GLM 5.3 and Qwen3.8 27B list tool support.

Tool calling is what turns a model into an agent. With open models on an OpenAI-compatible API, the request format is the same one you’d use with OpenAI. What differs is how often things go wrong and what the server does to prevent it. This post covers the format, how to write schemas open models handle well, parallel calls, the five failures you’ll actually see, and a Python loop that survives them.

How does tool calling work on an OpenAI-compatible API?

  1. You send messages plus a tools list. Each tool has a name, a description and a JSON Schema for its arguments.
  2. The model replies with finish_reason: "tool_calls" and a tool_calls list. Each entry has an id, a function name and arguments as a JSON string.
  3. Your code runs each call and appends one role: "tool" message per call, with the matching tool_call_id.
  4. You call the API again. The model answers or asks for more tools.

The model never runs anything. Your code decides what to execute.

How should you write tool schemas?

Open models follow simple schemas more reliably. These rules come from OpenAI’s guide and from what tends to break:

  • Name tools by action, such as read_file or search_code, and say in the description when to use each.
  • Keep the list short. OpenAI’s guide calls fewer than 20 functions at the start of a turn a soft suggestion. Every schema is also sent on every call; see the agent token budget.
  • Prefer flat arguments. Strings, numbers, booleans and enums. Avoid deep nesting and free-form objects.
  • Use enums for fixed choices, so the model can’t invent a value. The JSON schema generator turns a sample arguments object into a flat schema you can paste into parameters.
  • Mark required fields. OpenAI’s strict mode requires additionalProperties: false and every field in required, with null as a type for optional ones. Whether a provider enforces strict varies, so validate anyway.

What about parallel tool calls?

A model may return several entries in tool_calls in one turn, for example three file reads. Run independent calls concurrently and return one result per tool_call_id, in any order. If your tools have side effects that depend on each other, set parallel_tool_calls: false; OpenAI’s guide says that limits the model to zero or one call per turn. Test that your provider honors it.

Why do tool calls fail?

Failure What you see Fix
Malformed arguments json.loads raises Return the parse error as a tool result; retry
Schema mismatch Missing field, wrong type, invented enum Validate; return the validation error
Unknown tool A name you never defined Return “unknown tool, available: …”
No call when one is needed Text answer instead tool_choice: "required" or a named function
Cut-off call finish_reason: "length" Raise max_tokens; reasoning shares it

The first two are expected, not rare. vLLM’s tool-calling docs say that in auto mode, unless a tool sets strict: true (or the server enforces a strictness floor) and the model’s parser supports constraints, “vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function’s parameter schema.” A named function uses vLLM’s structured-outputs backend. Your provider may serve models differently, so treat validation as mandatory.

The last row catches people using reasoning models. GLM 5.3 and Qwen3.8 27B think before they answer, and thinking counts toward max_tokens. Why replies come back empty or cut off explains how much room to leave.

A tool loop that validates and retries

import json, os
from openai import OpenAI
from jsonschema import validate, ValidationError

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"], max_retries=4)
MODEL = "broadnet/glm-5.3"

TOOLS = [{"type": "function", "function": {
    "name": "get_order_status",
    "description": "Look up one order by its ID. Use when the user asks where an order is.",
    "parameters": {"type": "object",
                   "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"], "additionalProperties": False}}}]
SCHEMAS = {t["function"]["name"]: t["function"]["parameters"] for t in TOOLS}
IMPL = {"get_order_status": lambda order_id: f"Order {order_id}: shipped"}

def run_call(call):
    name = call.function.name
    if name not in IMPL:
        return f"Error: unknown tool {name}. Available: {', '.join(IMPL)}"
    try:
        args = json.loads(call.function.arguments or "{}")
        validate(args, SCHEMAS[name])
    except (json.JSONDecodeError, ValidationError) as e:
        return f"Error: invalid arguments ({e}). Send the call again with valid JSON."
    return str(IMPL[name](**args))

def agent(messages, max_steps=8):
    for _ in range(max_steps):
        r = client.chat.completions.create(model=MODEL, messages=messages,
                                           tools=TOOLS, max_tokens=4000)
        msg = r.choices[0].message
        if r.choices[0].finish_reason == "length":
            raise RuntimeError("Output limit reached; raise max_tokens")
        messages.append(msg.model_dump(exclude_none=True))
        if not msg.tool_calls:
            return msg.content
        for call in msg.tool_calls:
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": run_call(call)})
    raise RuntimeError("Step limit reached")

Three design choices matter here:

  1. Errors go back to the model as tool results. The model sees what was wrong and usually fixes it on the next turn. Raising an exception throws that chance away.
  2. The loop is bounded. max_steps stops a model that keeps calling tools without progress.
  3. Transport retries are separate. max_retries on the client handles 429 and 5xx. Bad arguments are handled in the loop. Don’t mix the two; see errors and retries.

For tools with side effects, such as sending an email, add an idempotency key so a retried step doesn’t act twice. Long-running agents covers this.

Which BroadNet models support tool calling?

Model Tools listed Context Notes
GLM 5.3 Yes 1,048,576 Also structured output
Qwen3.8 27B Yes 262,144 Also image input
DeepSeek V4.1 Flash Not listed 1,048,576 Image input; test tools before relying on them
Gemma 4 VL 26B Not listed 32,768 Not for tool loops

To choose between the two tool-capable models, see GLM 5.3 vs Qwen3.8 27B. When the final answer must match a shape but no function needs to run, structured output is simpler than a tool.

Where BroadNet Inference fits

BroadNet Inference serves GLM 5.3 and Qwen3.8 27B with tool calling listed, through the standard chat-completions format, so the loop above runs unchanged with the OpenAI Python SDK. Prices per token are on the pricing page. It doesn’t host server-side tools such as web search or code execution, and its docs don’t list tool calling for DeepSeek V4.1 Flash, the model on the unlimited plans. For a tool-heavy agent, use GLM 5.3 or Qwen3.8 27B per token. If you want the flat plan anyway, check on a $1 Meteor hour that DeepSeek V4.1 Flash calls your tools reliably before buying a month.

Questions

Do open models support function calling?

Many do, but support depends on the model and how the provider serves it. On BroadNet Inference, the docs list tool calling for GLM 5.3 and Qwen3.8 27B. They don't list it for DeepSeek V4.1 Flash or Gemma 4 VL 26B.

Why are my tool call arguments invalid JSON?

Unless the server constrains decoding, arguments are generated as text and parsed afterwards. vLLM's docs say arguments extracted from raw text may occasionally be malformed. Catch the parse error and send it back as a tool result so the model can correct itself.

How do I turn off parallel tool calls?

In the OpenAI format, set parallel_tool_calls to false. That limits the model to zero or one tool call per turn. Check that your provider honors the field.

Why does the model answer in text instead of calling a tool?

With tool_choice auto, the model decides. Set tool_choice to required, or to a specific function, when a call is mandatory. Also check that the tool description says when to use it.

Why is my tool call cut off?

Reasoning models count thinking toward max_tokens. A small limit can end the reply in the middle of the arguments. Check finish_reason and raise max_tokens.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity