BroadRouter
Get started

Prefill and decode

Updated

Prefill and decode are the two phases of LLM inference. In prefill the model processes the whole prompt in parallel and fills its KV cache; in decode it generates the reply one token at a time, reusing that cache. Prefill mostly sets time to first token; decode sets output speed.

Also called: prefill, prefill phase, decode phase, prefill stage, prompt processing

A chat request does two different jobs: read the prompt, then write the answer. Inference engineers call them prefill and decode, and they behave very differently on a GPU.

How it works

  1. Prefill. The model reads every prompt token in one parallel pass and stores each token’s attention keys and values in the KV cache. Nothing is returned yet.
  2. Decode. The model generates one token, appends its keys and values to the cache, and repeats until it stops or hits max_tokens.

The Sarathi-Serve paper puts the contrast plainly: “Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt”, while “decode iterations have low latency but also low compute utilization.”

The two phases map onto the two latency numbers users feel. The DistServe paper names them: “time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase.”

A worked example: a request with a 20,000-token prompt and a 300-token answer does all 20,000 tokens of prefill before the first streamed token appears, then 300 sequential decode steps. Cutting the prompt in half shortens the wait for the first token; it does not change how fast the 300 tokens arrive.

Why it matters

  • Time to first token tracks prompt size. Long system prompts, documents and tool lists all add prefill work. Cached prefixes skip some of it, which is why cached input tokens can make long prompts start faster.
  • Tokens per second is a decode number. Output speed depends on how quickly each decode step runs, which speculative decoding tries to raise.
  • The phases interfere. When a server runs both on the same GPU, a big prefill can pause other users’ decodes. DistServe reports “strong prefill-decoding interferences” and moves the phases to different GPUs. Sarathi-Serve instead “splits a prefill request into near equal sized chunks” and schedules them “without pausing ongoing decodes”.
  • Engines expose the trade-off. vLLM’s tuning guide says chunked prefill lets it batch prefill chunks with decode requests. A smaller max_num_batched_tokens improves inter-token latency, while a larger one favours TTFT and throughput.

The latency, TTFT and throughput guide shows how to measure both phases from the client side.

On BroadNet Inference

BroadNet does not publish how its servers schedule prefill and decode. What you can see is the effect: stream a response as in the streaming guide, log the time to the first content chunk (mostly prefill plus queueing) and the rate of chunks after it (decode). On GLM 5.3, DeepSeek V4.1 Flash and Qwen3.8 27B, reasoning tokens are generated in decode too, so they delay the visible answer and count toward max_tokens.

Common mistakes

  • Blaming a slow first token on the model’s output speed. Check prompt length first.
  • Benchmarking with a tiny prompt. Real prompts with long context have a much longer prefill.

Questions

Why does a long prompt make the first token slow?

Every prompt token has to go through prefill before the model can write anything. A longer prompt means more prefill work, so time to first token grows with prompt length unless part of the prompt is served from cache.

Does a longer answer slow down the first token?

No. The first token waits on prefill of the prompt. A longer answer only adds more decode steps after the first token has arrived.

What is chunked prefill?

A scheduling method that splits a long prompt into pieces and mixes them with ongoing decode steps, so one large prompt does not stall other users. vLLM enables it by default where it can.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity