TTFT is the pause before anything happens. In a chat window it is the blank screen; in a voice bot it is dead air.
How it works
NVIDIA’s benchmarking guide breaks TTFT into three parts: “request queuing time, prefill time, and network latency.” Each has its own cause:
| Part | What drives it | What you can change |
|---|---|---|
| Network | Distance, TLS setup, proxies | Reuse connections; call from a nearby region |
| Queueing | How busy the server is | Off-peak timing, a less loaded model or tier |
| Prefill | Prompt length, model size | Shorter prompts, stable cacheable prefixes |
Prefill is usually the big one for long inputs. The model runs the whole prompt through every layer to build the KV cache before it can produce token one. NVIDIA notes that on a busy server one request’s prefill can overlap with another’s generation, so TTFT also moves with other people’s traffic.
Two definitions matter when you read numbers. NVIDIA’s AIPerf counts the first token only if it carries content. For streaming text in the Responses API, OpenAI’s guide times TTFT “until you receive the first nonempty response.output_text.delta”. With reasoning tokens, the first visible word can come much later than the first generated token, so say which one you mean.
Why it matters
TTFT sets how fast a product feels, more than raw speed does. OpenAI’s guide says “a long prompt can delay the first token, especially when much of the input is new”. This is where cached input tokens help. The same guide says prompt caching “can reuse an eligible exact prefix match” and suggests comparing TTFT with and without cache hits. A cached prefix skips part of prefill, so a repeated system prompt costs less and can start sooner.
For agents, TTFT is paid on every step. Twenty tool calls with a two-second TTFT add forty seconds of waiting before counting any output.
TTFT is one part of end-to-end latency. The rest is generation time, set by tokens per second.
On BroadNet Inference
BroadNet’s docs describe streaming with Server-Sent Events, ending with data: [DONE], which is how you observe the first token on the client. DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, so time to first visible text includes thinking. BroadNet does not publish a TTFT figure. For a timing script and p50/p95 reporting, see LLM latency: TTFT vs tokens per second.