A chat request does two different jobs: read the prompt, then write the answer. Inference engineers call them prefill and decode, and they behave very differently on a GPU.
How it works
- Prefill. The model reads every prompt token in one parallel pass and stores each token’s attention keys and values in the KV cache. Nothing is returned yet.
- Decode. The model generates one token, appends its keys and values to the cache, and repeats until it stops or hits
max_tokens.
The Sarathi-Serve paper puts the contrast plainly: “Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt”, while “decode iterations have low latency but also low compute utilization.”
The two phases map onto the two latency numbers users feel. The DistServe paper names them: “time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase.”
A worked example: a request with a 20,000-token prompt and a 300-token answer does all 20,000 tokens of prefill before the first streamed token appears, then 300 sequential decode steps. Cutting the prompt in half shortens the wait for the first token; it does not change how fast the 300 tokens arrive.
Why it matters
- Time to first token tracks prompt size. Long system prompts, documents and tool lists all add prefill work. Cached prefixes skip some of it, which is why cached input tokens can make long prompts start faster.
- Tokens per second is a decode number. Output speed depends on how quickly each decode step runs, which speculative decoding tries to raise.
- The phases interfere. When a server runs both on the same GPU, a big prefill can pause other users’ decodes. DistServe reports “strong prefill-decoding interferences” and moves the phases to different GPUs. Sarathi-Serve instead “splits a prefill request into near equal sized chunks” and schedules them “without pausing ongoing decodes”.
- Engines expose the trade-off. vLLM’s tuning guide says chunked prefill lets it batch prefill chunks with decode requests. A smaller
max_num_batched_tokensimproves inter-token latency, while a larger one favours TTFT and throughput.
The latency, TTFT and throughput guide shows how to measure both phases from the client side.
On BroadNet Inference
BroadNet does not publish how its servers schedule prefill and decode. What you can see is the effect: stream a response as in the streaming guide, log the time to the first content chunk (mostly prefill plus queueing) and the rate of chunks after it (decode). On GLM 5.3, DeepSeek V4.1 Flash and Qwen3.8 27B, reasoning tokens are generated in decode too, so they delay the visible answer and count toward max_tokens.
Common mistakes
- Blaming a slow first token on the model’s output speed. Check prompt length first.
- Benchmarking with a tiny prompt. Real prompts with long context have a much longer prefill.