---
title: "Tool calling with open models - schemas, retries, code"
description: "Function calling on OpenAI-compatible open models: schema tips, parallel tool calls, the five ways tool calls fail, and a Python loop that validates and retries."
url: https://broadrouter.com/blog/tool-calling-open-models/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Tool calling with open models: schemas, parallel calls, retries

By the BroadNet Inference team · 4 min read · Published 2026-11-14 · Updated 2026-11-16

Open models on OpenAI-compatible APIs use the same tools and tool\_calls format as OpenAI. Keep schemas small and flat, parse arguments as JSON strings, validate them before running anything, and return errors to the model as tool results so it can retry. Cap the steps. On BroadNet, GLM 5.3 and Qwen3.8 27B list tool support.

## Key takeaways

-   The wire format is the same as OpenAI's: a tools list in, tool\_calls out, then a role tool message per tool\_call\_id.
-   Arguments arrive as a JSON string. vLLM's docs warn that when it extracts tool calls from raw text, they "may occasionally be malformed or violate the function's parameter schema".
-   Validate every call, send errors back as tool results, and cap retries and total steps.
-   Fewer, flatter tools work better. OpenAI suggests fewer than 20 functions at the start of a turn.
-   On BroadNet Inference, use GLM 5.3 or Qwen3.8 27B for tool calling; test DeepSeek V4.1 Flash before relying on it.

**On this page**

1.  How does tool calling work on an OpenAI-compatible API?
2.  How should you write tool schemas?
3.  What about parallel tool calls?
4.  Why do tool calls fail?
5.  A tool loop that validates and retries
6.  Which BroadNet models support tool calling?
7.  Where BroadNet Inference fits

[Tool calling](https://broadrouter.com/glossary/tool-calling/) is what turns a model into an agent. With open models on an [OpenAI-compatible API](https://broadrouter.com/glossary/openai-compatible-api/), the request format is the same one you’d use with OpenAI. What differs is how often things go wrong and what the server does to prevent it. This post covers the format, how to write schemas open models handle well, parallel calls, the five failures you’ll actually see, and a Python loop that survives them.

## How does tool calling work on an OpenAI-compatible API?

1.  You send `messages` plus a `tools` list. Each tool has a name, a description and a JSON Schema for its arguments.
2.  The model replies with `finish_reason: "tool_calls"` and a `tool_calls` list. Each entry has an `id`, a function name and `arguments` as a **JSON string**.
3.  Your code runs each call and appends one `role: "tool"` message per call, with the matching `tool_call_id`.
4.  You call the API again. The model answers or asks for more tools.

The model never runs anything. Your code decides what to execute.

## How should you write tool schemas?

Open models follow simple schemas more reliably. These rules come from OpenAI’s guide and from what tends to break:

-   **Name tools by action**, such as `read_file` or `search_code`, and say in the description _when_ to use each.
-   **Keep the list short.** OpenAI’s guide calls fewer than 20 functions at the start of a turn a soft suggestion. Every schema is also sent on every call; see [the agent token budget](https://broadrouter.com/blog/agent-token-budget/).
-   **Prefer flat arguments.** Strings, numbers, booleans and enums. Avoid deep nesting and free-form objects.
-   **Use enums** for fixed choices, so the model can’t invent a value. The [JSON schema generator](https://broadrouter.com/tools/json-schema-generator/) turns a sample arguments object into a flat schema you can paste into `parameters`.
-   **Mark required fields.** OpenAI’s strict mode requires `additionalProperties: false` and every field in `required`, with `null` as a type for optional ones. Whether a provider enforces `strict` varies, so validate anyway.

## What about parallel tool calls?

A model may return several entries in `tool_calls` in one turn, for example three file reads. Run independent calls concurrently and return one result per `tool_call_id`, in any order. If your tools have side effects that depend on each other, set `parallel_tool_calls: false`; OpenAI’s guide says that limits the model to zero or one call per turn. Test that your provider honors it.

## Why do tool calls fail?

| Failure | What you see | Fix |
| --- | --- | --- |
| Malformed arguments | `json.loads` raises | Return the parse error as a tool result; retry |
| Schema mismatch | Missing field, wrong type, invented enum | Validate; return the validation error |
| Unknown tool | A name you never defined | Return “unknown tool, available: …” |
| No call when one is needed | Text answer instead | `tool_choice: "required"` or a named function |
| Cut-off call | `finish_reason: "length"` | Raise `max_tokens`; reasoning shares it |

The first two are expected, not rare. vLLM’s tool-calling docs say that in `auto` mode, unless a tool sets `strict: true` (or the server enforces a strictness floor) and the model’s parser supports constraints, “vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function’s parameter schema.” A named function uses vLLM’s structured-outputs backend. Your provider may serve models differently, so treat validation as mandatory.

The last row catches people using [reasoning models](https://broadrouter.com/glossary/reasoning-tokens/). GLM 5.3 and Qwen3.8 27B think before they answer, and thinking counts toward [`max_tokens`](https://broadrouter.com/glossary/max-tokens/). [Why replies come back empty or cut off](https://broadrouter.com/guides/reasoning-tokens-empty-response/) explains how much room to leave.

## A tool loop that validates and retries

```python
import json, os
from openai import OpenAI
from jsonschema import validate, ValidationError

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"], max_retries=4)
MODEL = "broadnet/glm-5.3"

TOOLS = [{"type": "function", "function": {
    "name": "get_order_status",
    "description": "Look up one order by its ID. Use when the user asks where an order is.",
    "parameters": {"type": "object",
                   "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"], "additionalProperties": False}}}]
SCHEMAS = {t["function"]["name"]: t["function"]["parameters"] for t in TOOLS}
IMPL = {"get_order_status": lambda order_id: f"Order {order_id}: shipped"}

def run_call(call):
    name = call.function.name
    if name not in IMPL:
        return f"Error: unknown tool {name}. Available: {', '.join(IMPL)}"
    try:
        args = json.loads(call.function.arguments or "{}")
        validate(args, SCHEMAS[name])
    except (json.JSONDecodeError, ValidationError) as e:
        return f"Error: invalid arguments ({e}). Send the call again with valid JSON."
    return str(IMPL[name](**args))

def agent(messages, max_steps=8):
    for _ in range(max_steps):
        r = client.chat.completions.create(model=MODEL, messages=messages,
                                           tools=TOOLS, max_tokens=4000)
        msg = r.choices[0].message
        if r.choices[0].finish_reason == "length":
            raise RuntimeError("Output limit reached; raise max_tokens")
        messages.append(msg.model_dump(exclude_none=True))
        if not msg.tool_calls:
            return msg.content
        for call in msg.tool_calls:
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": run_call(call)})
    raise RuntimeError("Step limit reached")
```

Three design choices matter here:

1.  **Errors go back to the model as tool results.** The model sees what was wrong and usually fixes it on the next turn. Raising an exception throws that chance away.
2.  **The loop is bounded.** `max_steps` stops a model that keeps calling tools without progress.
3.  **Transport retries are separate.** `max_retries` on the client handles 429 and 5xx. Bad arguments are handled in the loop. Don’t mix the two; see [errors and retries](https://broadrouter.com/guides/errors-and-retries/).

For tools with side effects, such as sending an email, add an idempotency key so a retried step doesn’t act twice. [Long-running agents](https://broadrouter.com/blog/long-running-agents-timeouts-retries/) covers this.

## Which BroadNet models support tool calling?

| Model | Tools listed | Context | Notes |
| --- | --- | --- | --- |
| [GLM 5.3](https://broadrouter.com/models/glm-5.3/) | Yes | 1,048,576 | Also structured output |
| [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) | Yes | 262,144 | Also image input |
| [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) | Not listed | 1,048,576 | Image input; test tools before relying on them |
| [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/) | Not listed | 32,768 | Not for tool loops |

To choose between the two tool-capable models, see [GLM 5.3 vs Qwen3.8 27B](https://broadrouter.com/compare/glm-5.3-vs-qwen3.8-27b/). When the final answer must match a shape but no function needs to run, [structured output](https://broadrouter.com/blog/structured-output-json-schema/) is simpler than a tool.

## Where BroadNet Inference fits

BroadNet Inference serves GLM 5.3 and Qwen3.8 27B with tool calling listed, through the standard chat-completions format, so the loop above runs unchanged with the [OpenAI Python SDK](https://broadrouter.com/guides/openai-python-sdk/). Prices per token are on the [pricing page](https://broadrouter.com/pricing/). It doesn’t host server-side tools such as web search or code execution, and its docs don’t list tool calling for DeepSeek V4.1 Flash, the model on the unlimited plans. For a tool-heavy agent, use GLM 5.3 or Qwen3.8 27B per token. If you want the flat plan anyway, check on a $1 Meteor hour that DeepSeek V4.1 Flash calls your tools reliably before buying a month.

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[Open-weight model](https://broadrouter.com/glossary/open-weight-model/)

An open-weight model is a machine learning model whose trained weights are published for anyone to download and run, usually under a licence with conditions. It differs from fully open-source AI, which also releases the training code and detailed information about the training data, so others can study and reproduce the model.

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[vLLM](https://broadrouter.com/glossary/vllm/)

vLLM is an open-source library for LLM inference and serving, first developed at UC Berkeley's Sky Computing Lab. It manages KV cache memory with PagedAttention, batches incoming requests continuously, supports many quantization formats and exposes an OpenAI-compatible HTTP server.

[JSON Schema](https://broadrouter.com/glossary/json-schema/)

JSON Schema is a vocabulary, written in JSON, for describing and validating the structure of JSON data: types, required fields, allowed values and nesting. The current version is 2020-12. LLM APIs use it to define tool parameters and the exact shape a structured output must follow.

[OpenAI-compatible API](https://broadrouter.com/glossary/openai-compatible-api/)

An OpenAI-compatible API is an HTTP API that accepts the same request and response formats as the OpenAI API, such as POST /v1/chat/completions, so OpenAI SDKs and tools can use it by changing only the base URL, the API key and the model name.

[Structured output](https://broadrouter.com/glossary/structured-output/)

Structured output is an LLM API feature that constrains a model's reply to a format the developer specifies, usually a JSON Schema, so code can parse it reliably. It is stricter than JSON mode, which guarantees valid JSON but not a particular shape.

[Idempotency key](https://broadrouter.com/glossary/idempotency-key/)

An idempotency key is a unique value, usually a random ID such as a UUID, that a client attaches to a request that changes state, such as a payment or a purchase. If the same request is sent again with the same key, the server returns the original result instead of performing the action twice.

## Questions

### Do open models support function calling?

Many do, but support depends on the model and how the provider serves it. On BroadNet Inference, the docs list tool calling for GLM 5.3 and Qwen3.8 27B. They don't list it for DeepSeek V4.1 Flash or Gemma 4 VL 26B.

### Why are my tool call arguments invalid JSON?

Unless the server constrains decoding, arguments are generated as text and parsed afterwards. vLLM's docs say arguments extracted from raw text may occasionally be malformed. Catch the parse error and send it back as a tool result so the model can correct itself.

### How do I turn off parallel tool calls?

In the OpenAI format, set parallel\_tool\_calls to false. That limits the model to zero or one tool call per turn. Check that your provider honors the field.

### Why does the model answer in text instead of calling a tool?

With tool\_choice auto, the model decides. Set tool\_choice to required, or to a specific function, when a call is mandatory. Also check that the tool description says when to use it.

### Why is my tool call cut off?

Reasoning models count thinking toward max\_tokens. A small limit can end the reply in the middle of the arguments. Check finish\_reason and raise max\_tokens.

## Keep reading

-   [OpenClaw and Hermes Agent on a cheap model: setup and token math](https://broadrouter.com/blog/openclaw-hermes-agent-cheap-model/)
-   [JSON mode vs json\_schema: structured output that holds up](https://broadrouter.com/blog/structured-output-json-schema/)
-   [Use MCP servers with OpenAI-compatible models (any provider)](https://broadrouter.com/blog/mcp-with-openai-compatible-models/)
-   [Long-running agents: timeouts, retries and checkpoints that hold](https://broadrouter.com/blog/long-running-agents-timeouts-retries/)
-   [Agent token budget: where an agent loop's tokens go](https://broadrouter.com/blog/agent-token-budget/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)
-   [Guide: Run the OpenAI Agents SDK on BroadNet Inference](https://broadrouter.com/guides/openai-agents-sdk-custom-provider/)
-   [Guide: Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)
-   [Guide: BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-tool-calling-open-models&utm_content=page-cta) ·

**Sources**

-   [OpenAI, function calling guide (checked 2026-11-14)](https://developers.openai.com/api/docs/guides/function-calling)
-   [vLLM docs, tool calling (checked 2026-11-14)](https://docs.vllm.ai/en/latest/features/tool_calling/)
-   [BroadNet Inference API reference, models](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-tool-calling-open-models&utm_content=sources#models)
