---
title: "LLM latency explained - TTFT, tokens/sec, end-to-end"
description: "What time to first token, tokens per second and end-to-end latency measure, how to time each from the client with code, and which fixes move which number."
url: https://broadrouter.com/blog/llm-latency-ttft-throughput/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# LLM latency: TTFT vs tokens per second vs end-to-end time

By the BroadNet Inference team · 4 min read · Published 2026-11-14 · Updated 2026-11-16

Time to first token (TTFT) is the wait before the first output token. Tokens per second is how fast the answer then streams. End-to-end latency is the total: TTFT plus generation time, roughly output tokens divided by per-user speed. Prompt length and queueing drive TTFT, model size and load drive speed, and answer length drives the total. Measure all three at p50 and p95.

## Key takeaways

-   End-to-end latency ≈ TTFT + (output tokens − 1) × inter-token latency, roughly TTFT + output tokens ÷ per-user tokens per second. For long answers, speed and length dominate; for short ones, TTFT does.
-   On reasoning models, the first visible token comes after the thinking, so measure time to first content token separately from time to first token.
-   Long prompts raise TTFT because the model reads the whole prompt first; cached prefixes usually help.
-   At about 100 tokens per second, generation time grows in step with answer length: ten times the output tokens takes about ten times as long to stream.
-   Report p50 and p95 with the prompt length; an average without prompt size is not comparable.

**On this page**

1.  What are TTFT, tokens per second and end-to-end latency?
2.  Why do reasoning models need a fourth number?
3.  How do you measure latency from the client?
4.  What actually improves each number?
5.  How does prompt length change the trade-off?
6.  Where BroadNet Inference fits

“The API is slow” can mean three different things, and each has a different fix. A chat user waits for the first word. An agent waits for the whole tool call. A batch job cares about tokens per hour. This post defines the three numbers, shows how to measure them from your own code, and maps each fix to the number it actually moves.

## What are TTFT, tokens per second and end-to-end latency?

NVIDIA’s benchmarking guide gives clean definitions, and they match what you can observe from a client:

| Metric | What it measures | Who feels it | Driven by |
| --- | --- | --- | --- |
| [Time to first token](https://broadrouter.com/glossary/time-to-first-token/) (TTFT) | Request sent → first token received | Chat users | Network, queueing, prompt length (prefill), cache hits |
| Inter-token latency (ITL) | Average gap between tokens | Readers of a streamed answer | Model size, hardware, server load |
| [Tokens per second](https://broadrouter.com/glossary/tokens-per-second/) per user | Output tokens after the first ÷ generation time (first token to last); equals 1 ÷ average ITL | Readers, agents | Same as ITL |
| End-to-end latency | Request sent → last token received | Agents, APIs, batch | TTFT + generation time |

The formula that ties them together:

**end-to-end ≈ TTFT + (output tokens − 1) × ITL ≈ TTFT + output tokens ÷ per-user tokens per second**

Per-user speed is not the same as server [throughput](https://broadrouter.com/glossary/throughput/), the total tokens a server produces across all users; a provider can raise one while the other falls.

Take the typical speed BroadNet publishes for its plan model, about 100 tokens per second. Generation time scales with answer length, so a 4,000-token code file takes about eighty times as long to generate as a 50-token classification, while both wait the same TTFT. For short answers TTFT is most of the wait; for long ones, speed and length are.

## Why do reasoning models need a fourth number?

Reasoning models such as [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) think before they answer, and the thinking is output tokens too. If the model spends 800 tokens reasoning at about 100 tokens per second, the user waits for all 800 to be generated before the first visible word, even if TTFT on the wire looks fast. So measure **time to first content token** separately. On BroadNet Inference, `reasoning_effort: "low"` shortens the thinking for simple steps; the [reasoning guide](https://broadrouter.com/guides/reasoning-tokens-empty-response/) explains the trade-off.

## How do you measure latency from the client?

Time it where your users are, with [streaming](https://broadrouter.com/glossary/streaming/) on. Server-side timers miss the network and any proxy.

```python
import os, time, statistics
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

def timed(model, messages, max_tokens=1500):
    t0 = time.perf_counter()
    first = first_content = None
    usage = None
    stream = client.chat.completions.create(
        model=model, messages=messages, max_tokens=max_tokens,
        stream=True, stream_options={"include_usage": True},
    )
    for chunk in stream:
        now = time.perf_counter()
        if chunk.choices:
            if first is None:
                first = now
            if chunk.choices[0].delta.content and first_content is None:
                first_content = now
        if chunk.usage:
            usage = chunk.usage
    end = time.perf_counter()
    out = usage.completion_tokens if usage else 0
    return {
        "ttft": first - t0,
        "ttfct": (first_content or end) - t0,
        "e2e": end - t0,
        "tps": (out - 1) / (end - first) if out > 1 and end > first else 0,
        "prompt_tokens": usage.prompt_tokens if usage else None,
    }

runs = [timed("broadnet/gemma-4-26b-a4b", [{"role": "user", "content": "Summarise SSE in 100 words."}]) for _ in range(50)]
for k in ("ttft", "ttfct", "e2e", "tps"):
    xs = sorted(r[k] for r in runs)
    print(k, "p50", round(statistics.median(xs), 3), "p95", round(xs[int(len(xs) * 0.95) - 1], 3))
```

Five rules for a fair benchmark:

1.  **Fix the prompt length** and report it. TTFT at 500 prompt tokens and at 200,000 are different numbers. The [token counter](https://broadrouter.com/tools/token-counter/) tells you how long a test prompt is before you send it.
2.  **Fix the output length** with `max_tokens` and a prompt that fills it, so tokens per second is comparable.
3.  **Report p50 and p95**, not the mean. Tail latency is what users remember.
4.  **Spread runs over time.** Load changes during the day.
5.  **Warm up first.** The first call pays for DNS, TLS and connection setup; reuse one client.

## What actually improves each number?

| To improve | Do this | Why |
| --- | --- | --- |
| TTFT | Shorten or cache the prompt prefix | Prefill time grows with prompt length; cached prefixes skip work |
| TTFT | Keep connections warm, use a nearby region | Removes setup and round trips |
| Time to first content token | Lower reasoning effort, or use a non-reasoning model for simple steps | Less hidden thinking before the answer |
| Tokens per second | Use a smaller or faster model for the step | Decode speed depends on model size and serving |
| End-to-end | Ask for shorter output (schemas, word limits) | Output length multiplies ITL |
| End-to-end for many calls | Run independent calls in parallel within your [concurrency](https://broadrouter.com/glossary/concurrency/) limit | Wall-clock time drops even though each call is the same |
| Perceived latency | Stream to the user | They read while the model writes |

Output length is the most underrated lever. Asking for “JSON with these four fields” instead of “explain your answer” can cut generation time by an order of magnitude with no infrastructure work. [Structured output](https://broadrouter.com/glossary/structured-output/) helps here.

The cache lever is covered in [prompt structure for caching](https://broadrouter.com/blog/prompt-structure-for-caching/). The parallelism lever runs into rate limits quickly; see [handling 429 errors](https://broadrouter.com/blog/handle-llm-api-429-rate-limits/).

## How does prompt length change the trade-off?

TTFT grows with the prompt because the model must process every input token before it can write. That is the hidden cost of very long prompts and part of the [long context vs RAG](https://broadrouter.com/blog/long-context-vs-rag/) decision: a 1M-token prompt answers questions RAG cannot, but it starts slower. If interactive speed matters, keep the hot path short and send the big context only when the question needs it.

## Where BroadNet Inference fits

BroadNet publishes a typical speed of about 100 tokens per second for its plan model and streams with the standard SSE format, so the code above works unchanged. It does not publish TTFT figures or latency SLAs, so measure on your own prompts before you commit; a $1 [Meteor hour](https://inference.broadnet.ai/plans?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-latency-ttft-throughput&utm_content=body) is enough for a 50-run benchmark, and the starting credit covers one on Gemma 4 VL 26B. If you need guaranteed sub-second latency at large scale, test carefully: a pay-as-you-go key allows 4 requests in flight, which suits apps and agents more than high-volume real-time serving. Models and their prices are on the [models](https://broadrouter.com/models/) and [pricing](https://broadrouter.com/pricing/) pages.

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Time to first token (TTFT)](https://broadrouter.com/glossary/time-to-first-token/)

Time to first token (TTFT) is the time from sending a request to a language model until the first output token arrives. It includes network time, queueing on the server and prefill, the step where the model reads the whole prompt, so it grows with prompt length and load.

[Latency](https://broadrouter.com/glossary/latency/)

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

[Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/)

Input tokens are the tokens a language model reads in a request, including the system prompt, conversation history, tool definitions and images. Output tokens are the tokens it generates in reply, including any reasoning. LLM APIs usually price them separately, and output usually costs more.

[Tokens per second](https://broadrouter.com/glossary/tokens-per-second/)

Tokens per second (TPS) is the rate at which a language model generates output tokens, usually measured for one request after its first token arrives. It sets how fast a reply streams once it starts. Summed across all concurrent requests it becomes system throughput; the wait before the first token is measured separately.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Prefill and decode](https://broadrouter.com/glossary/prefill-and-decode/)

Prefill and decode are the two phases of LLM inference. In prefill the model processes the whole prompt in parallel and fills its KV cache; in decode it generates the reply one token at a time, reusing that cache. Prefill mostly sets time to first token; decode sets output speed.

## Questions

### What is a good time to first token for an LLM API?

For chat, under about one second on short prompts feels responsive. Long prompts and reasoning models take longer by design. Compare TTFT only at the same prompt length and model.

### Is tokens per second the same as throughput?

Not quite. Per-user tokens per second is how fast one answer streams. Throughput is the total tokens a server produces across all users. A provider can raise total throughput while each user's speed drops.

### Why is TTFT so high on a reasoning model?

The model generates hidden reasoning tokens before the visible answer. The first token may arrive fast, but the first content token waits until thinking is done. Lower reasoning\_effort for simple tasks.

### Does streaming reduce latency?

It does not shorten end-to-end time. It lets the user read the answer while it is still being written, so perceived latency drops to roughly the TTFT.

### How many requests do I need for a reliable latency benchmark?

Enough to see the tail. Fifty to a hundred requests per prompt size gives a usable p95. Spread them across the day, because load changes.

## Keep reading

-   [LLM streaming explained: SSE chunks, deltas, usage and cancel](https://broadrouter.com/blog/llm-streaming-sse-explained/)
-   [How to structure prompts so the prefix caches](https://broadrouter.com/blog/prompt-structure-for-caching/)
-   [Long context vs RAG: when 1M tokens replaces a vector DB](https://broadrouter.com/blog/long-context-vs-rag/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: Stream chat completions and handle dropped connections](https://broadrouter.com/guides/streaming-chat-completions/)
-   [Guide: Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-latency-ttft-throughput&utm_content=page-cta) ·

**Sources**

-   [NVIDIA NIM benchmarking guide, LLM latency and throughput metrics (fetched 2026-11-14)](https://docs.nvidia.com/nim/benchmarking/llm/metrics)
-   [BroadNet Inference plans, typical speed (checked 2026-11-15)](https://inference.broadnet.ai/plans?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-latency-ttft-throughput&utm_content=sources)
-   [BroadNet Inference API reference, streaming and reasoning (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-llm-latency-ttft-throughput&utm_content=sources)
-   [Baseten Inference Engineering, measuring latency and throughput (checked 2026-11-16)](https://www.baseten.co/inference-engineering/book/01-prerequisites/measuring-latency-and-throughput/)
