BroadRouter
Get started

When a 1M-token context window is worth it (and when not)

By the BroadNet Inference team · 4 min read · Published

A 1M-token context window lets one request hold a whole codebase, a contract set or hours of transcripts, so the model sees everything at once instead of retrieved fragments. It pays off for cross-file reasoning and one-off analysis; for repeated lookups over a large, changing corpus, retrieval is cheaper. DeepSeek V4.1 Flash and GLM 5.3 accept 1,048,576 tokens.

Light shining through towering cliffs made of stacked paper pages.

Context windows have grown from a few thousand tokens to over a million. DeepSeek V4.1 Flash and GLM 5.3 on BroadNet Inference accept 1,048,576 tokens per request, prompt and answer together. That changes how you can build, but it doesn’t make every prompt better.

What fits in a million tokens?

As a rough guide: 1,000 words of English prose came to about 1,290 tokens, averaged across five tokenizers, in our tokenizer efficiency dataset, so a million tokens is roughly 810,000 words of plain prose:

Content Roughly
A mid-sized codebase (tens of thousands of lines) Fits, with room to spare
Several long contracts with schedules Fits
A day of meeting transcripts Fits
A large monorepo with dependencies Often doesn’t; select directories

Token counts depend on language and content. Code and Arabic text often use more tokens per character than English prose. Measure with the usage block on a real request rather than estimating, or check a sample in the context window calculator.

Which BroadNet models have a 1M-token context window?

Model Context window Input Output Images
DeepSeek V4.1 Flash 1,048,576 $0.175 $0.52 Yes
GLM 5.3 1,048,576 $0.78 $3.46 No
Qwen3.8 27B 262,144 $0.175 $2.17 Yes
BroadNet Gemma 4 VL 26B 32,768 $0.10 $0.22 Yes

Prices are per 1M tokens. GLM 5.3 adds tool calling and structured output on the same 1M window.

When is a 1M-token window worth it?

  • Cross-file reasoning. “Why does this test fail?” often depends on five files nobody would think to retrieve together.
  • Whole-document review. Finding conflicts between clause 4 and schedule B needs both in view.
  • One-off analysis. For a question you’ll ask once, building a retrieval index costs more effort than it saves.
  • Agents. A sub-agent that can read the whole module makes fewer wrong guesses. See running parallel agents.

When is retrieval still the better choice?

Retrieval wins for many small lookups over a library far bigger than any window, for content that changes every minute, and for fast interactive answers where reading a huge prompt first adds delay. Many production systems do both: retrieve the most relevant 200,000 tokens, then let a long-context model reason across all of them. The full decision, with cost per query, latency and recall side by side, is in long context vs RAG.

Does the model really use the whole window?

Not evenly. The research is consistent on this:

  • The Lost in the Middle paper (Liu et al.) found that performance “is often highest when relevant information occurs at the beginning or end of the input context”.
  • Chroma’s Context Rot report tested 18 models and found that “performance grows increasingly unreliable as input length grows”, even on simple tasks.

These studies did not test the models on BroadNet, so treat them as a warning, not a measurement. The practical rule: a window that accepts a million tokens does not mean every token gets equal attention. Put the most important material near the start or the end, and test recall on your own documents.

What does a million-token prompt cost?

On a pay-as-you-go key, DeepSeek V4.1 Flash bills $0.175 per 1M input tokens and $0.0055 per 1M cached input tokens. A full million-token prompt costs about $0.175 the first time, and much less on repeat calls when the prefix is cached. How to raise your cache rate.

The prompt caching calculator shows how much a cache hit saves on your own prompt sizes. A worked example: ten questions about the same 800,000-token contract set. Without caching, that is 8M tokens at $0.175 per 1M. If the documents come first and stay identical, the first call reads 800,000 new tokens and the next nine can read most of them from cache at $0.0055 per 1M. Only the question and the answer change.

On an unlimited plan the size of the prompt does not change the bill: Full Moon is $60 a month per session with no token meter. That is why long-context agents are a common reason to choose a flat plan over per-token billing.

How much room should you leave for the answer?

The window covers prompt and output, and on reasoning models the output includes thinking. If you send 1,040,000 tokens of prompt, only about 8,000 remain for reasoning and answer. Keep a margin, set max_tokens explicitly, and read why reasoning models can return an empty answer.

How to prepare a codebase for a 1M-token prompt

  1. Exclude what the model doesn’t need. Lock files, build output, vendored dependencies, images and generated code take tokens and add noise.
  2. Count before you send. Send a small sample, read prompt_tokens from usage, and scale up.
  3. Order by stability. Rarely changed files first, the files under edit last, the question at the very end. That keeps the prefix cacheable.
  4. Label every file. Wrap files in clear markers such as === path/to/file.py === so the model can cite them.
  5. Ask for citations. “Quote the line you rely on” reduces confident mistakes over huge inputs.
  6. Measure. Log prompt tokens, cached tokens and time to first token per request. Streaming shows the first token as soon as it arrives.

Where BroadNet Inference fits

Two of its chat models take 1,048,576 tokens per request, and BroadNet does not trim your context. On a per-token key, caching keeps repeated long prompts affordable; on a plan session, prompt size does not affect the bill. If your corpus is far larger than a million tokens or changes every minute, you still need retrieval, and no window size changes that.

Questions

Does BroadNet trim long prompts?

No. BroadNet does not trim your context. If the prompt plus max_tokens exceeds the model's window, shorten the prompt rather than relying on silent truncation.

How many words is 1 million tokens?

It depends on the language and content. English prose measured about 129 tokens per 100 words in BroadRouter's tokenizer dataset (1,000 words came to about 1,290 tokens, averaged across five tokenizers), so 1,048,576 tokens hold roughly 810,000 words, well over a thousand pages. Technical text, code and Arabic need more tokens, so count your real content.

Is a 1M-token request slow?

It takes longer to start than a short request, because the model must read the whole prompt first. Cached prefixes usually help on repeat calls. Measure time to first token on your own prompts.

Do LLMs actually use the whole context window?

Not evenly. The Lost in the Middle paper found accuracy is often highest when the relevant information is at the start or end of the input, and Chroma's Context Rot report found reliability falls as input length grows. Put key material at the edges and test on your own data.

Should I stop using RAG?

Not necessarily. Use long context when the model needs to see everything together, and retrieval when it needs a few facts from a huge library. Many systems combine both.

What happens if my prompt is bigger than the context window?

BroadNet does not trim it for you, so the request will not run as intended. Remove files, summarise old turns, or retrieve only the relevant parts before you send.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity