You need a token count before you send a request for three reasons: to stay inside the context window, to pick a sensible max_tokens, and to estimate cost. Exact counts come only from the model’s own tokenizer. Good estimates come from calibration. This post shows both, with real measurements.
Why do token counts differ between models?
A tokenizer splits text into pieces from a fixed vocabulary. Each model family trains its own. A larger vocabulary, or one trained on more of a given language, produces fewer tokens for that language. So “how many tokens is this prompt” has a different answer for each model.
We measured the same short customer-service message, translated into four languages, plus a small Python function and an invoice JSON object. The OpenAI counts come from tiktoken (cl100k_base, GPT-4 era, and o200k_base, GPT-4o era). The open-model counts come from the tokenizer files each family publishes on Hugging Face, counted without the chat template, on 14 November 2026.
| Text (same meaning) | Characters | cl100k_base | o200k_base | DeepSeek V4.1 Flash | GLM 5.3 | Qwen3.8 27B | Gemma 4 |
|---|---|---|---|---|---|---|---|
| English, 31 words | 173 | 37 | 37 | 37 | 38 | 39 | 39 |
| Spanish | 160 | 47 | 40 | 50 | 45 | 43 | 42 |
| Arabic | 142 | 102 | 46 | 55 | 58 | 48 | 49 |
| Chinese | 54 | 51 | 37 | 34 | 35 | 36 | 38 |
| Python function | 94 | 33 | 33 | 35 | 33 | 35 | 38 |
| Invoice JSON | 140 | 59 | 60 | 63 | 60 | 72 | 72 |
One sample per row, so treat the numbers as an illustration, not a benchmark. Three lessons hold anyway. The tokenizer generation matters: the Arabic text more than halved from cl100k_base to o200k_base. Open models differ from each other too: the same Arabic text ranged from 48 to 58 tokens, and the JSON from 60 to 72. And structured content is token-dense everywhere: the JSON ran at about 2 characters per token on every tokenizer.
You can repeat this on your own text with the free token counter, or the per-family counters for DeepSeek, Qwen and GPT. To see what an edit to a prompt saves, compare both versions in the prompt token diff. Why Arabic in particular costs more tokens, measured across longer texts, is covered in tokens per word in Arabic vs English.
How do you count tokens exactly?
If the model’s tokenizer is public, load it and count locally. For OpenAI models:
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
n = len(enc.encode(text))
For open-weight families, the tokenizer usually ships with the model on Hugging Face and loads with AutoTokenizer.from_pretrained(...). Two caveats: the chat template adds tokens for roles and separators, and an API provider’s deployed version may differ from the public files. BroadNet’s API reference does not document a token-counting endpoint or the tokenizer for each model, so the public tokenizer gives a close estimate and the exact number is usage.prompt_tokens after the call.
How do you estimate tokens when the tokenizer is not public?
Calibrate once per model, then estimate locally:
import os
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
def prompt_tokens(model, text):
r = client.chat.completions.create(
model=model, max_tokens=1000, # room for reasoning on thinking models
messages=[{"role": "user", "content": text}],
)
return r.usage.prompt_tokens
def measure(model, samples: dict[str, list[str]]):
base = prompt_tokens(model, "hi") # chat-template overhead plus one short word
ratios = {}
for kind, texts in samples.items():
chars = toks = 0
for t in texts:
chars += len(t)
toks += prompt_tokens(model, t) - base
ratios[kind] = chars / toks
return ratios # e.g. {"english": 4.1, "arabic": 2.9, "json": 2.3}
def estimate(text, kind, ratios, overhead=10):
return int(len(text) / ratios[kind]) + overhead
Use 20 or more real samples per content type (your actual prompts, not lorem ipsum). The one-word baseline request removes the fixed chat-template overhead, which estimate adds back as overhead; set it to your measured baseline. Calibrate on BroadNet’s cheapest model only if it is the one you will call: Gemma 4 VL 26B ratios do not transfer to DeepSeek V4.1 Flash. Images are billed as input tokens too, and larger images use more, so calibrate them separately by resolution.
What margins should you use for max_tokens and context?
The window covers prompt and output. On reasoning models the output includes hidden thinking, reported as reasoning tokens. Budget like this:
- Prompt estimate = calibrated count × 1.15 (a 15 percent safety margin).
- Answer budget = the longest answer you accept.
- Reasoning budget = for thinking models, at least 1,000 tokens; BroadNet’s docs recommend at least 1,000 or
reasoning_effort: "low", because a small limit can return an empty answer. - max_tokens = answer budget + reasoning budget.
- Check: prompt estimate + max_tokens ≤ context window. If not, cut the prompt, not the answer.
Context windows on BroadNet Inference, from the published price file:
| Model | Context window | Practical prompt ceiling with 16K output and 15% margin |
|---|---|---|
| DeepSeek V4.1 Flash | 1,048,576 | about 897,000 |
| GLM 5.3 | 1,048,576 | about 897,000 |
| Qwen3.8 27B | 262,144 | about 213,000 |
| Gemma 4 VL 26B | 32,768 | about 14,000 |
The ceiling column is (window − 16,384) ÷ 1.15, rounded down. Change 16,384 to your own max_tokens.
What goes wrong with token estimates?
- Counting words instead of tokens. The 0.75-words-per-token rule is for English. Our 29-word Arabic sample was 46 to 102 tokens depending on the tokenizer.
- Forgetting tools and system prompt. Tool schemas are JSON, which is token-dense. Count them as part of the prompt.
- Assuming silent truncation. Some tools drop old messages without telling you. BroadNet says it does not trim your context, so check the size before you send rather than relying on the server to cut it. The context window calculator does the check for you.
- Ignoring history growth. In a chat or agent loop, the prompt grows every turn. Re-check the budget each call, not once per session. Stable history also caches better; see prompt structure for caching.
If you work in Arabic, the tokenizer is one more reason to test models on your own text; choosing an LLM for Arabic covers the rest. To turn token counts into a monthly bill, see how to estimate LLM API cost before launch.
Where BroadNet Inference fits
BroadNet Inference returns prompt_tokens, completion_tokens and, when the model reports them, cached and reasoning token counts in every response, which is all you need to calibrate. It does not publish a counting endpoint or per-model tokenizers, so pre-send counts are estimates. Long-context work on the 1M-token models is covered in the context window guide, and prices per 1M tokens are on the pricing page.
