BroadRouter
Get started

RAG chunk size and overlap: what they do to retrieval and cost

By the BroadNet Inference team · 7 min read · Published · Updated

Start with chunks of about 256 to 512 tokens, 0 to 15% overlap, and top-k tuned on your own queries. Bigger chunks raise the tokens every query sends to the LLM. Overlap multiplies the tokens you embed and store by size ÷ (size − overlap), so 50% overlap doubles the index. Published tests found small or no-overlap chunks often recall as well or better.

Chunk size and overlap are the two numbers every RAG pipeline sets early and rarely revisits. They decide how many vectors you store, how many tokens you embed, how many tokens every question sends to the LLM, and whether the right passage is found at all. This post shows what published tests found, then works through the token math so you can see the cost of each setting before you embed anything.

If you are still deciding whether to use RAG at all, start with long context vs RAG. For Arabic documents, read Arabic RAG and Arabic text normalization alongside this post.

What do chunk size and overlap change?

  • Chunk size is how many tokens go into each piece you embed and retrieve.
  • Overlap is how many tokens consecutive chunks share, so a sentence cut at a boundary appears whole in at least one chunk.
  • Top-k is how many chunks you retrieve per question.

Together they set four things:

Quantity Formula (long documents)
Chunks stored corpus tokens ÷ (size − overlap)
Tokens embedded corpus tokens × size ÷ (size − overlap)
Context per query top-k × size
LLM input per query top-k × size + system prompt + question

The first two are paid once per indexing run, and again every time you re-chunk. The last is paid on every question, forever.

What do published tests say about chunk size?

Two public evaluations are worth knowing, because they measured different things.

Chroma (Smith and Troynikov, July 2024) measured retrieval at the token level: how much of the relevant text was retrieved (recall), and how much of what was retrieved was relevant. It used five retrieved chunks and OpenAI’s text-embedding-3-large. Selected rows:

Splitter Size (tokens) Overlap Recall IoU
Recursive 800 400 85.4% 1.5%
Token 800 400 87.9% 1.4%
Recursive 400 200 88.1% 3.3%
Recursive 400 0 89.5% 3.6%
Recursive 200 0 88.1% 6.9%

IoU (intersection over union) measures how much of the retrieved text was relevant, so it doubles as a measure of wasted tokens. The 800/400 rows match OpenAI’s default for file search; assuming a token splitter, Chroma found that setting gave slightly below-average recall and the lowest scores on the other metrics. Smaller chunks with no overlap recalled as much or more and carried far less irrelevant text.

NVIDIA (June 2025) measured end-to-end answer accuracy on five datasets, with token chunks of 128 to 2,048 at 15% overlap, plus page-level and section-level chunking. Page-level chunking had the highest average accuracy (0.648) and the lowest variance. The best token size depended on the questions: factoid sets favoured 256 to 512 tokens, analytical ones 1,024 tokens or whole pages. Both extremes, 128 and 2,048, usually did worse.

The two agree on one thing: there is no universal best size, and very large chunks with heavy overlap are rarely the right default.

How much does overlap inflate the index?

Each chunk advances by size − overlap tokens, so the index holds size ÷ (size − overlap) copies of the corpus. For a 2,000,000-token corpus of 10,000-token documents:

Size Overlap Chunks Tokens embedded Inflation
200 0 10,000 2,000,000 1.00×
400 0 5,000 2,000,000 1.00×
512 77 (15%) 4,600 2,338,800 1.17×
1,024 154 (15%) 2,400 2,338,800 1.17×
800 400 (50%) 4,800 3,840,000 1.92×
400 200 (50%) 9,800 3,920,000 1.96×

Short documents inflate a little less, because the last chunk of each one is partial. The formula gives the limit: 15% overlap costs about 18% more, and 50% costs about 100% more, in embedding tokens, vector storage and re-indexing time.

Overlap also costs at query time. With 50% overlap, two neighbouring chunks retrieved for the same question repeat half their text. You pay for those duplicate tokens, and they push other relevant passages out of top-k.

BroadNet doesn’t sell embeddings, so this side of the bill goes to your embedding provider. Price it with the embedding cost calculator.

What does chunk size do to the LLM bill?

Every question sends top-k chunks plus the prompt. Take 100,000 questions a month, 600 tokens of system prompt and question, and the same 300-token answer each time:

Size × top-k Context per query Input per query Input per month
200 × 10 2,000 2,600 260 million
400 × 5 2,000 2,600 260 million
512 × 5 2,560 3,160 316 million
800 × 5 4,000 4,600 460 million
1,024 × 5 5,120 5,720 572 million
1,024 × 8 8,192 8,792 879 million

Output is 30 million tokens a month in every row, so only input changes. At a price per 1M tokens, monthly input cost is the millions in the last column times that price. On Gemma 4 VL 26B at $0.10 per 1M input tokens, the 1,024 × 8 setup costs 879 × $0.10 a month in input against 260 × $0.10 for 400 × 5. That is 3.4 times as much, for a setting the published tests don’t show recalling better. On Qwen3.8 27B ($0.175 input) or GLM 5.3 ($0.78 input), the ratio is the same and the totals are larger.

Two levers move this more than the model price does:

  1. Lower k when you raise the size. 200 × 10 and 400 × 5 send the same context. Keep context per query roughly fixed while you test sizes, so you compare retrieval quality, not context budget.
  2. Cache the fixed part. Put the system prompt first and keep it byte-identical, so it can bill at a cached input rate on models that have one. Retrieved chunks change per question and rarely cache.

Context limits matter too. Gemma 4 VL 26B accepts 32,768 tokens per request, plenty for any row above. If you push k × size toward tens of thousands of tokens, you have moved from RAG towards long context, and that comparison has its own math. The RAG chunking calculator runs these formulas for your own corpus, chunk size, overlap and top-k.

How do you chunk by tokens without cutting words?

Count in tokens, but cut at structure. LangChain’s RecursiveCharacterTextSplitter measures size with its length_function, which is character length by default. A 512 “size” is then 512 characters, roughly 100 to 150 tokens of English, and a different number again for Arabic; see tokens per word in Arabic. Embedding limits and LLM bills are in tokens.

This chunker packs whole paragraphs up to a token budget and carries trailing paragraphs forward as overlap. Cutting at paragraph boundaries avoids splitting words or multi-byte Arabic characters, which raw token windows can do.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")    # an estimate: count with your embedder's tokenizer
count = lambda s: len(enc.encode(s))

def chunk(text, size=512, overlap=64):
    """Pack whole paragraphs up to `size` tokens; repeat trailing paragraphs up to `overlap` tokens."""
    chunks, cur, cur_len = [], [], 0
    for para in (p.strip() for p in text.split("\n\n")):
        if not para:
            continue
        n = count(para)
        if cur and cur_len + n > size:
            chunks.append("\n\n".join(cur))
            tail, t = [], 0
            for q in reversed(cur):                  # carry the end of this chunk forward
                if t + count(q) > overlap:
                    break
                tail.insert(0, q); t += count(q)
            cur, cur_len = tail, t
        cur.append(para); cur_len += n              # a paragraph over `size` stays whole
    if cur:
        chunks.append("\n\n".join(cur))
    return chunks

def plan(corpus_tokens, size, overlap, k, queries_per_month, prompt_tokens=600):
    index_tokens = corpus_tokens * size / (size - overlap)   # long-document limit
    per_query = k * size + prompt_tokens
    return {"chunks": round(index_tokens / size), "tokens_to_embed": round(index_tokens),
            "input_per_query": per_query, "input_per_month": per_query * queries_per_month}

print(plan(2_000_000, 512, 64, 5, 100_000))
# {'chunks': 4464, 'tokens_to_embed': 2285714, 'input_per_query': 3160, 'input_per_month': 316000000}

Overlap here is “up to”: a paragraph longer than the overlap budget is not repeated. A single paragraph longer than size stays whole; split such paragraphs at sentences first if your documents have them. Prefix each chunk with its document title and section heading, so a retrieved chunk still says what it is about. That adds a few tokens per chunk and often helps more than overlap.

How do you pick the numbers for your data?

  1. Build an evaluation set. 50 to 200 real questions, each with the passage that answers it. The LLM evaluation guide covers scoring.
  2. Fix the context budget. Hold k × size near one value, such as 2,000 tokens, while you compare sizes.
  3. Try three sizes and two overlaps. For example 256, 512 and 1,024 tokens, each at 0 and 15% overlap. Measure recall of the right passage at k.
  4. Try structure. If your documents have pages or numbered sections, test page-level or section-level chunks against the token sizes. NVIDIA found page-level the most consistent.
  5. Pick the cheapest setting within a point or two of the best recall, then price it at your query volume with the monthly cost method.

Do this on a sample before you embed the full corpus. Changing chunking later means re-embedding everything.

Where BroadNet Inference fits

BroadNet Inference serves the answer step of RAG, not retrieval. It sells no embeddings or vector storage, so pair it with an embeddings provider and a vector database of your choice. For the answer, Gemma 4 VL 26B is the low-cost option and runs on the $5 starting credit. Qwen3.8 27B (262,144-token context) and GLM 5.3 (1,048,576) suit larger top-k or long documents, after a top-up. Prompts and retrieved passages are not retained after the request. If your corpus is small and stable enough to fit in the context window, long context may beat chunking altogether.

Questions

What is a good chunk size for RAG?

Start around 256 to 512 tokens and test. Chroma's evaluation found 200 to 400 tokens with no overlap performed well. NVIDIA found 256 to 512 tokens suited factoid questions, while analytical questions did better with 1,024-token or page-level chunks.

How much chunk overlap should I use?

Less than most defaults. NVIDIA found 15% best of the 10%, 15% and 20% it tried on one dataset. Chroma found no-overlap settings often scored as well or better. Every point of overlap adds tokens to embed, store and send.

Does a bigger chunk size cost more?

Not to embed, without overlap: the corpus is the same number of tokens. It costs more per query, because each retrieved chunk is longer, so top-k × chunk size grows. Lower k when you raise the size.

Is chunk size measured in tokens or characters?

It depends on the splitter. LangChain's RecursiveCharacterTextSplitter measures with its length_function, which is character length by default. Embedding limits and LLM bills are in tokens, so pass a token counter.

Do I have to re-embed when I change chunk size?

Yes. New boundaries make new chunks, and each needs a new vector. Re-embedding costs the whole index again, so settle chunking on a sample before you embed the full corpus.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity