BroadRouter
Get started

Long context vs RAG: when 1M tokens replaces a vector DB

By the BroadNet Inference team · 5 min read · Published · Updated

Long context replaces RAG when the corpus fits in the window, changes rarely and questions need many parts of it at once; caching keeps repeat queries cheap. RAG still wins when the corpus outgrows the window, updates constantly, needs per-user permissions or must answer in a second or two. Many systems retrieve with RAG and reason with long context.

The 1M-token context window guide covers what fits in a million tokens and what one request costs. This post answers a narrower question that comes up in every architecture review: can we delete the vector database and the RAG pipeline around it? Sometimes yes. Here is how to decide, on four axes: cost, latency, recall and upkeep.

When does long context replace RAG?

All four of these should be true:

  1. The corpus fits. Prompt plus answer plus reasoning stays inside the context window with a margin. On DeepSeek V4.1 Flash and GLM 5.3 that window is 1,048,576 tokens (the two compared). The context window calculator checks whether your documents fit.
  2. It changes rarely. A manual updated monthly is fine. A ticket queue updated every minute is not.
  3. Everyone may see all of it. One shared corpus, no per-user permissions to filter.
  4. Questions span the corpus. “Which contracts conflict with the new policy?” needs everything in view; “what is the refund window?” does not.

If one is false, keep retrieval, or use a hybrid.

How do cost per query compare?

Take a 400,000-token document set and a question with a 500-token answer. RAG retrieves, say, 8,000 tokens of chunks. On a pay-as-you-go key with DeepSeek V4.1 Flash:

Approach Input per query Input cost per query
Long context, first query 400,000 new 0.4 × $0.175
Long context, repeat query, prefix cached about 400,000 cached about 0.4 × $0.0055
RAG 8,000 new 0.008 × $0.175

Output is the same in each case: 500 tokens × $0.52 per 1M. (Each multiplier is millions of tokens times the price per 1M.)

Three things follow. Without caching, long context costs 50 times RAG per query on input, because it sends 50 times the tokens. With the corpus cached, the gap shrinks a lot, since cached input on this model costs a few percent of new input. But caching only holds while the prefix stays byte-identical, so the corpus must come first and stay stable; see prompt structure for caching. RAG has its own costs that this table leaves out: embedding the corpus, running the vector store, and the engineering time to tune chunking. The embedding cost calculator prices the embedding part. The RAG chunking calculator prices the query side: chunks, tokens per query and the monthly LLM bill.

To see what your own cache rate is worth, try the prompt caching calculator. On a flat plan the input column disappears: Full Moon is $60 a month per session with no token meter, so a 400,000-token prompt costs the same as a short one.

How does latency compare?

A model reads the whole prompt before it writes the first token. Time to first token grows with prompt length, and a prompt hundreds of thousands of tokens long takes noticeably longer to start than an 8,000-token one. Cached prefixes help in many serving systems. Vector search itself usually adds little; the retrieval call is small next to generation.

So for an interactive assistant that must reply in a second or two, RAG is the safer default. For an analyst who asks one deep question and waits for the answer, long context is fine. Measure TTFT on your own prompts at the size you plan to send.

Which has better recall?

Neither is perfect, and they fail in different ways.

Long context can lose facts in the middle (lost in the middle). The Lost in the Middle study found models used information best when it sat at the start or end of the input, and worse in between. The RULER benchmark found that models scoring near-perfect on simple needle-in-a-haystack tests degraded on harder tasks as inputs grew, and that of the models claiming 32K tokens or more, “only half of them” kept satisfactory performance at 32K. Newer models do better, but the advertised window is not the same as the window where quality holds.

RAG fails before the model starts: if the retriever misses the passage, the model never sees it. Questions that need many passages, or that use different words from the source, are the classic misses.

Failure mode Long context RAG
Fact buried mid-prompt Risk grows with length Low, prompt is short
Answer needs 20 scattered passages Strong Weak unless top-k is large
Query wording differs from the source Strong Depends on embeddings and reranking
Corpus bigger than the window Not possible Strong

Two mitigations for long context: put the question at the end, after the documents, and ask the model to quote the lines it relies on.

What about upkeep?

RAG is a system: chunking, embeddings, an index, re-indexing on change, permission filters, evaluation of retrieval quality. Long context is a prompt: concatenate files with clear labels and send. For a small team with a corpus that fits, deleting the pipeline removes real work. For a large, changing, permissioned corpus, the pipeline is doing a job long context cannot.

The hybrid most teams end up with

  1. Retrieve generously, not five chunks but whole documents, up to a few hundred thousand tokens.
  2. Order them stably (by document ID, not by score) so the prefix caches across related questions.
  3. Put the question last and send it to a long-context model.
  4. Ask for citations to document labels, and spot-check them.

This keeps RAG’s reach over a huge corpus and long context’s ability to reason across many sources. It also tolerates a weaker retriever, because a near-miss document is still in the prompt.

Where BroadNet Inference fits

BroadNet Inference offers two 1M-token models, DeepSeek V4.1 Flash and GLM 5.3, and it does not trim your context, so an oversized request fails rather than losing documents silently. Cached input is billed at the lower cached rate on a per-token key, and the unlimited plans remove the token meter entirely for DeepSeek V4.1 Flash. Its API reference lists chat completions and transcription, not an embeddings endpoint or a vector store, so for the RAG half you bring your own. Leave room for reasoning tokens in the window; the reasoning guide explains why. Prices are on the pricing page.

Questions

Is RAG dead now that models have 1M-token context windows?

No. Long context removes the need for retrieval when the whole corpus fits and is stable. Most company knowledge bases are larger than a million tokens, change daily and need permissions, which is still RAG territory.

Is long context more accurate than RAG?

It depends on the question. Long context does better when the answer needs many passages combined. RAG can do better on single-fact lookups in huge corpora, because a short prompt keeps the model focused. Test both on real questions.

How many words fit in a 1 million token context window?

About 810,000 words of plain English prose in DeepSeek V4.1 Flash's 1,048,576-token window, at the 1,290 tokens per 1,000 words measured across five tokenizers in BroadRouter's tokenizer dataset. Technical English, code, tables and Arabic need more tokens per word, so less fits. Leave room for the answer and reasoning.

Can I combine RAG and long context?

Yes, and it is often the best option. Retrieve generously, for example 100,000 to 200,000 tokens of candidate documents, then let a long-context model reason over all of them instead of the top five chunks.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity