---
title: "Count tokens before sending - estimates, margins, languages"
description: "Estimate prompt tokens by language and model before calling an LLM API: why tokenizers differ, measured examples in five languages, and safe max_tokens margins."
url: https://broadrouter.com/blog/count-tokens-before-sending/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# How to count tokens before you send an LLM request

By the BroadNet Inference team · 5 min read · Published 2026-11-14 · Updated 2026-11-16

Count tokens with the tokenizer of the model you call when it is public; otherwise calibrate a characters-per-token ratio from the usage field of real requests, per language and content type. Add a 10 to 20 percent margin, then check that prompt tokens plus max\_tokens fit the context window, leaving room for reasoning tokens on thinking models.

## Key takeaways

-   Tokenizers differ by model. The same Arabic sentence measured 102 tokens with cl100k\_base, 46 with o200k\_base and 48 to 58 on the open models' tokenizers in our test.
-   English prose ran at about 4.5 characters per token on every tokenizer in this short test; code, JSON and most non-Latin scripts use more tokens per character.
-   When a model's tokenizer is not public, calibrate: send samples, read usage.prompt\_tokens, and store a ratio per language and content type.
-   The context window covers prompt plus output. Budget prompt + max\_tokens + a margin, never prompt alone.
-   On reasoning models, thinking counts toward max\_tokens; BroadNet recommends at least 1,000 or reasoning\_effort low.

**On this page**

1.  Why do token counts differ between models?
2.  How do you count tokens exactly?
3.  How do you estimate tokens when the tokenizer is not public?
4.  What margins should you use for max\_tokens and context?
5.  What goes wrong with token estimates?
6.  Where BroadNet Inference fits

You need a [token](https://broadrouter.com/glossary/token/) count before you send a request for three reasons: to stay inside the [context window](https://broadrouter.com/glossary/context-window/), to pick a sensible [max\_tokens](https://broadrouter.com/glossary/max-tokens/), and to estimate cost. Exact counts come only from the model’s own tokenizer. Good estimates come from calibration. This post shows both, with real measurements.

## Why do token counts differ between models?

A tokenizer splits text into pieces from a fixed vocabulary. Each model family trains its own. A larger vocabulary, or one trained on more of a given language, produces fewer tokens for that language. So “how many tokens is this prompt” has a different answer for each model.

We measured the same short customer-service message, translated into four languages, plus a small Python function and an invoice JSON object. The OpenAI counts come from [tiktoken](https://github.com/openai/tiktoken) (`cl100k_base`, GPT-4 era, and `o200k_base`, GPT-4o era). The open-model counts come from the tokenizer files each family publishes on Hugging Face, counted without the chat template, on 14 November 2026.

| Text (same meaning) | Characters | cl100k\_base | o200k\_base | DeepSeek V4.1 Flash | GLM 5.3 | Qwen3.8 27B | Gemma 4 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| English, 31 words | 173 | 37 | 37 | 37 | 38 | 39 | 39 |
| Spanish | 160 | 47 | 40 | 50 | 45 | 43 | 42 |
| Arabic | 142 | 102 | 46 | 55 | 58 | 48 | 49 |
| Chinese | 54 | 51 | 37 | 34 | 35 | 36 | 38 |
| Python function | 94 | 33 | 33 | 35 | 33 | 35 | 38 |
| Invoice JSON | 140 | 59 | 60 | 63 | 60 | 72 | 72 |

One sample per row, so treat the numbers as an illustration, not a benchmark. Three lessons hold anyway. The tokenizer generation matters: the Arabic text more than halved from `cl100k_base` to `o200k_base`. Open models differ from each other too: the same Arabic text ranged from 48 to 58 tokens, and the JSON from 60 to 72. And structured content is token-dense everywhere: the JSON ran at about 2 characters per token on every tokenizer.

You can repeat this on your own text with the free [token counter](https://broadrouter.com/tools/token-counter/), or the per-family counters for [DeepSeek](https://broadrouter.com/tools/deepseek-token-counter/), [Qwen](https://broadrouter.com/tools/qwen-token-counter/) and [GPT](https://broadrouter.com/tools/gpt-token-counter/). To see what an edit to a prompt saves, compare both versions in the [prompt token diff](https://broadrouter.com/tools/prompt-token-diff/). Why Arabic in particular costs more tokens, measured across longer texts, is covered in [tokens per word in Arabic vs English](https://broadrouter.com/blog/tokens-per-word-arabic-english/).

## How do you count tokens exactly?

If the model’s tokenizer is public, load it and count locally. For OpenAI models:

```python
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
n = len(enc.encode(text))
```

For open-weight families, the tokenizer usually ships with the model on Hugging Face and loads with `AutoTokenizer.from_pretrained(...)`. Two caveats: the chat template adds tokens for roles and separators, and an API provider’s deployed version may differ from the public files. BroadNet’s API reference does not document a token-counting endpoint or the tokenizer for each model, so the public tokenizer gives a close estimate and the exact number is `usage.prompt_tokens` after the call.

## How do you estimate tokens when the tokenizer is not public?

Calibrate once per model, then estimate locally:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

def prompt_tokens(model, text):
    r = client.chat.completions.create(
        model=model, max_tokens=1000,  # room for reasoning on thinking models
        messages=[{"role": "user", "content": text}],
    )
    return r.usage.prompt_tokens

def measure(model, samples: dict[str, list[str]]):
    base = prompt_tokens(model, "hi")  # chat-template overhead plus one short word
    ratios = {}
    for kind, texts in samples.items():
        chars = toks = 0
        for t in texts:
            chars += len(t)
            toks += prompt_tokens(model, t) - base
        ratios[kind] = chars / toks
    return ratios  # e.g. {"english": 4.1, "arabic": 2.9, "json": 2.3}

def estimate(text, kind, ratios, overhead=10):
    return int(len(text) / ratios[kind]) + overhead
```

Use 20 or more real samples per content type (your actual prompts, not lorem ipsum). The one-word baseline request removes the fixed chat-template overhead, which `estimate` adds back as `overhead`; set it to your measured baseline. Calibrate on BroadNet’s cheapest model only if it is the one you will call: [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/) ratios do not transfer to [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/). Images are billed as input tokens too, and larger images use more, so calibrate them separately by resolution.

## What margins should you use for max\_tokens and context?

The window covers prompt **and** output. On reasoning models the output includes hidden thinking, reported as [reasoning tokens](https://broadrouter.com/glossary/reasoning-tokens/). Budget like this:

1.  **Prompt estimate** = calibrated count × 1.15 (a 15 percent safety margin).
2.  **Answer budget** = the longest answer you accept.
3.  **Reasoning budget** = for thinking models, at least 1,000 tokens; BroadNet’s docs recommend at least 1,000 or `reasoning_effort: "low"`, because a small limit can return an empty answer.
4.  **max\_tokens** = answer budget + reasoning budget.
5.  **Check**: prompt estimate + max\_tokens ≤ context window. If not, cut the prompt, not the answer.

Context windows on BroadNet Inference, from the published price file:

| Model | Context window | Practical prompt ceiling with 16K output and 15% margin |
| --- | --- | --- |
| DeepSeek V4.1 Flash | 1,048,576 | about 897,000 |
| [GLM 5.3](https://broadrouter.com/models/glm-5.3/) | 1,048,576 | about 897,000 |
| [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) | 262,144 | about 213,000 |
| Gemma 4 VL 26B | 32,768 | about 14,000 |

The ceiling column is (window − 16,384) ÷ 1.15, rounded down. Change 16,384 to your own max\_tokens.

## What goes wrong with token estimates?

-   **Counting words instead of tokens.** The 0.75-words-per-token rule is for English. Our 29-word Arabic sample was 46 to 102 tokens depending on the tokenizer.
-   **Forgetting tools and system prompt.** Tool schemas are JSON, which is token-dense. Count them as part of the prompt.
-   **Assuming silent truncation.** Some tools drop old messages without telling you. BroadNet says it does not trim your context, so check the size before you send rather than relying on the server to cut it. The [context window calculator](https://broadrouter.com/tools/context-window-calculator/) does the check for you.
-   **Ignoring history growth.** In a chat or agent loop, the prompt grows every turn. Re-check the budget each call, not once per session. Stable history also caches better; see [prompt structure for caching](https://broadrouter.com/blog/prompt-structure-for-caching/).

If you work in Arabic, the tokenizer is one more reason to test models on your own text; [choosing an LLM for Arabic](https://broadrouter.com/blog/choosing-an-llm-for-arabic/) covers the rest. To turn token counts into a monthly bill, see [how to estimate LLM API cost before launch](https://broadrouter.com/blog/estimate-llm-api-cost-before-launch/).

## Where BroadNet Inference fits

BroadNet Inference returns `prompt_tokens`, `completion_tokens` and, when the model reports them, cached and reasoning token counts in every response, which is all you need to calibrate. It does not publish a counting endpoint or per-model tokenizers, so pre-send counts are estimates. Long-context work on the 1M-token models is covered in the [context window guide](https://broadrouter.com/blog/million-token-context-window-guide/), and prices per 1M tokens are on the [pricing page](https://broadrouter.com/pricing/).

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Tokenizer](https://broadrouter.com/glossary/tokenizer/)

A tokenizer is the component that converts text into the sequence of integer token IDs a language model reads, and converts generated IDs back into text. Each model ships its own tokenizer, so the same text can produce a different number of tokens on different models.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[Context window](https://broadrouter.com/glossary/context-window/)

A context window is the maximum number of tokens a language model can work with in a single request. It covers the prompt, conversation history, tool definitions, images and the generated output, including any reasoning tokens. Anything that does not fit must be cut, summarised or split across several requests.

[max\_tokens](https://broadrouter.com/glossary/max-tokens/)

max\_tokens is an LLM API parameter that sets the most tokens a model may generate in one response. When generation reaches the limit, it stops and the response is marked as cut off, for example with finish\_reason set to "length".

[Chat template](https://broadrouter.com/glossary/chat-template/)

A chat template is the formatting rule, usually written in Jinja and shipped with a model's tokenizer, that converts a list of chat messages with roles into the single token sequence the model expects, including its special control tokens. Using the wrong template can sharply degrade a model's answers.

[AI agent](https://broadrouter.com/glossary/agent/)

An AI agent is a system in which a language model directs its own steps toward a goal: it decides which tools to call, reads the results and repeats until it produces a final answer or hits a limit. A fixed script that calls a model at set points is usually called a workflow instead.

[Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/)

Input tokens are the tokens a language model reads in a request, including the system prompt, conversation history, tool definitions and images. Output tokens are the tokens it generates in reply, including any reasoning. LLM APIs usually price them separately, and output usually costs more.

## Questions

### How many tokens is a word?

In BroadRouter's tokenizer dataset, which pools five tokenizers (those of the four BroadNet chat models and OpenAI's o200k\_base; the post's example table adds cl100k\_base), 1,000 English words is about 1,290 tokens, or about 1.29 tokens per word. The common rule of thumb, 0.75 words per token, is close for English prose. Arabic, other non-Latin scripts and code use more tokens per word. Measure your own text with the model you call.

### Can I use tiktoken to count tokens for DeepSeek, Qwen or Gemma?

Only as a rough estimate. tiktoken implements OpenAI encodings. Other model families ship their own tokenizers, often on Hugging Face, and the counts differ. Calibrate against the usage field.

### Is there a free tool to count tokens for DeepSeek, Qwen, GLM or Gemma?

Yes. BroadRouter's free token counter runs the published tokenizers for these model families in your browser, so the text does not leave your machine. Treat the count as an estimate of what the API will report, because chat templates add a few tokens.

### What happens if my prompt is longer than the context window?

Most APIs return an error. Some tools trim old messages silently. BroadNet does not trim your context, so check prompt plus max\_tokens against the window before sending.

### How much margin should I leave for max\_tokens?

Estimate the longest answer you expect, add the reasoning budget for thinking models, and keep 10 to 20 percent of the window free for estimation error.

## Keep reading

-   [Tokens per word in Arabic vs English: why Arabic costs more](https://broadrouter.com/blog/tokens-per-word-arabic-english/)
-   [When a 1M-token context window is worth it (and when not)](https://broadrouter.com/blog/million-token-context-window-guide/)
-   [How to estimate LLM API cost before you launch a feature](https://broadrouter.com/blog/estimate-llm-api-cost-before-launch/)
-   [How to structure prompts so the prefix caches](https://broadrouter.com/blog/prompt-structure-for-caching/)
-   [Reduce prompt tokens: measured cuts for system prompts, tools, data](https://broadrouter.com/blog/reduce-prompt-tokens/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-count-tokens-before-sending&utm_content=page-cta) ·

**Sources**

-   [tiktoken README, encodings and about 4 bytes per token (fetched 2026-11-14)](https://github.com/openai/tiktoken)
-   [BroadNet Inference API reference, max\_tokens and reasoning (fetched 2026-11-14)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-count-tokens-before-sending&utm_content=sources)
-   [BroadNet Inference machine-readable prices and context lengths (pricing.json, fetched 2026-11-14)](https://inference.broadnet.ai/pricing.json?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-count-tokens-before-sending&utm_content=sources)
-   [Hugging Face Transformers, tokenizer documentation (fetched 2026-11-14)](https://huggingface.co/docs/transformers/main_classes/tokenizer)
