Qwen token counter
Updated
In short: This counter loads the tokenizer.json from the Qwen/Qwen3.8-27B repository, so the count is what the model reads. Its vocabulary has 248,077 token IDs, one of the largest here, and it is the most compact of the five we measured on Modern Standard Arabic (about 183 tokens per 100 words).
tokens
| Model | Tokens | Chars per token | As input | As output |
|---|---|---|---|---|
| Qwen3.8 27B Qwen3.8 27B tokenizer | … | … | … | … |
| DeepSeek V4.1 Flash DeepSeek V4.1 Flash tokenizer | … | … | … | … |
| GLM 5.3 GLM 5.3 tokenizer | … | … | … | … |
| Gemma 4 VL 26B Gemma 4 tokenizer (Google base model) | … | … | … | … |
| GPT (o200k) OpenAI o200k_base | … | … | … | … |
Cost = tokens × the model's published price per 1M input or output tokens. GPT is shown for reference only (no BroadNet price). Measured ratios for every tokenizer are in the tokenizer efficiency dataset. Rows marked ≈ are estimates from measured characters per token until you load that tokenizer; loading downloads its tokenizer.json from Hugging Face once and keeps it in your browser cache.
Other counters: DeepSeek · GLM · Gemma · GPT · Llama 3 · Llama 4 · Kimi · Mistral Small · Ministral 3 · gpt-oss · Phi-4 · Gemma 3 · MiniMax · Nemotron 3 · Qwen3 · all models. Prices synced 2026-11-14.
Report a wrong price or number (email to info@broadnet.ai, with this page's link and your inputs filled in).
How it works
- Paste text into the box or pick a sample. The Qwen3.8 27B tokenizer is preselected and downloads once (13 MB) when you start.
- The text appears split into tokens, a colour per token; tick "Show token IDs" to see the numbers.
- Tick "Chat format" to include the system prompt and chat-template tokens a request adds.
- Compare the table with the other models and copy the link or download CSV.
About the Qwen tokenizer
Qwen3.8 uses a byte-level BPE tokenizer with 248,044 learned tokens, almost double the size of DeepSeek's. It applies NFC normalisation and keeps combining marks with their letters, which helps Arabic diacritics and Indic scripts. Unlike GLM and DeepSeek it splits numbers into single digits, so long numbers cost more tokens.
On Modern Standard Arabic sentences Qwen needs only 1.31× the tokens of the English translations, the lowest Arabic tax of the BroadNet models we measured. Its larger vocabulary means a 13 MB download the first time you use it.
| Tokenizer | Qwen3.8 27B tokenizer |
|---|---|
| Source | Qwen/Qwen3.8-27B (commit 3756181) |
| Token IDs | 248,077 (248,044 in the base vocabulary) |
| Context window | 262,144 tokens |
| Price per 1M tokens | $0.175 input · $2.17 output |
| English prose | 132 tokens per 100 words (rank 5 of 5) |
| Arabic | 183 tokens per 100 words (rank 1 of 5) |
| Python code | 215 tokens per 100 words (rank 4 of 5) |
| Download | 13 MB, once; then cached in your browser |
Tokens per 100 words measured on the open texts of our dataset with every tokenizer (2026-11-15; Arabic = Modern Standard Arabic); rank 1 = fewest tokens. All counts: tokenizer efficiency dataset. On Arabic the most compact was Qwen, on code GLM. Full table on the token counter.
The cost formula
input cost = tokens × input price per 1M ÷ 1,000,000
output cost = tokens × output price per 1M ÷ 1,000,000Worked example
The English sample in the counter is 260 words. The Qwen3.8 27B tokenizer splits it into 298 tokens. Sent to Qwen3.8 27B as a prompt it costs $0.000052; written by the model as an answer it costs $0.00065. Prices are per one million tokens, so multiply the count by the price and divide by 1,000,000.
Questions
Which tokenizer does Qwen3.8 27B use?
A byte-level BPE tokenizer with 248,077 token IDs, published as tokenizer.json in the Qwen/Qwen3.8-27B repository. This page loads that file, pinned to commit 3756181.
Why does Qwen use more tokens for numbers?
Its pre-tokenizer splits every digit into its own piece before merging, so 2026 is four tokens. DeepSeek and GLM group up to three digits. Number-heavy JSON or tables therefore cost a little more on Qwen.
How many tokens does the Qwen chat template add?
About 52 tokens for one user message with the official template's defaults, because the template writes a reasoning-effort instruction into the system turn. The <|im_start|> and <|im_end|> markers are one token each.
Is Qwen cheaper for Arabic?
Per token it is priced at $0.175 per 1M input and $2.17 per 1M output. Its tokenizer also needs fewer tokens for Arabic than DeepSeek or GLM in our measurements, so Arabic prompts cost less than the price alone suggests. BroadNet tested it in Arabic and Gulf and Egyptian dialects: Qwen3.8 27B.
How much fits in Qwen3.8's context window?
262,144 tokens, prompt and answer together: roughly 198.594 thousand English words. Use the context window calculator to check a file.