“The API is slow” can mean three different things, and each has a different fix. A chat user waits for the first word. An agent waits for the whole tool call. A batch job cares about tokens per hour. This post defines the three numbers, shows how to measure them from your own code, and maps each fix to the number it actually moves.
What are TTFT, tokens per second and end-to-end latency?
NVIDIA’s benchmarking guide gives clean definitions, and they match what you can observe from a client:
| Metric | What it measures | Who feels it | Driven by |
|---|---|---|---|
| Time to first token (TTFT) | Request sent → first token received | Chat users | Network, queueing, prompt length (prefill), cache hits |
| Inter-token latency (ITL) | Average gap between tokens | Readers of a streamed answer | Model size, hardware, server load |
| Tokens per second per user | Output tokens after the first ÷ generation time (first token to last); equals 1 ÷ average ITL | Readers, agents | Same as ITL |
| End-to-end latency | Request sent → last token received | Agents, APIs, batch | TTFT + generation time |
The formula that ties them together:
end-to-end ≈ TTFT + (output tokens − 1) × ITL ≈ TTFT + output tokens ÷ per-user tokens per second
Per-user speed is not the same as server throughput, the total tokens a server produces across all users; a provider can raise one while the other falls.
Take the typical speed BroadNet publishes for its plan model, about 100 tokens per second. Generation time scales with answer length, so a 4,000-token code file takes about eighty times as long to generate as a 50-token classification, while both wait the same TTFT. For short answers TTFT is most of the wait; for long ones, speed and length are.
Why do reasoning models need a fourth number?
Reasoning models such as DeepSeek V4.1 Flash think before they answer, and the thinking is output tokens too. If the model spends 800 tokens reasoning at about 100 tokens per second, the user waits for all 800 to be generated before the first visible word, even if TTFT on the wire looks fast. So measure time to first content token separately. On BroadNet Inference, reasoning_effort: "low" shortens the thinking for simple steps; the reasoning guide explains the trade-off.
How do you measure latency from the client?
Time it where your users are, with streaming on. Server-side timers miss the network and any proxy.
import os, time, statistics
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
def timed(model, messages, max_tokens=1500):
t0 = time.perf_counter()
first = first_content = None
usage = None
stream = client.chat.completions.create(
model=model, messages=messages, max_tokens=max_tokens,
stream=True, stream_options={"include_usage": True},
)
for chunk in stream:
now = time.perf_counter()
if chunk.choices:
if first is None:
first = now
if chunk.choices[0].delta.content and first_content is None:
first_content = now
if chunk.usage:
usage = chunk.usage
end = time.perf_counter()
out = usage.completion_tokens if usage else 0
return {
"ttft": first - t0,
"ttfct": (first_content or end) - t0,
"e2e": end - t0,
"tps": (out - 1) / (end - first) if out > 1 and end > first else 0,
"prompt_tokens": usage.prompt_tokens if usage else None,
}
runs = [timed("broadnet/gemma-4-26b-a4b", [{"role": "user", "content": "Summarise SSE in 100 words."}]) for _ in range(50)]
for k in ("ttft", "ttfct", "e2e", "tps"):
xs = sorted(r[k] for r in runs)
print(k, "p50", round(statistics.median(xs), 3), "p95", round(xs[int(len(xs) * 0.95) - 1], 3))
Five rules for a fair benchmark:
- Fix the prompt length and report it. TTFT at 500 prompt tokens and at 200,000 are different numbers. The token counter tells you how long a test prompt is before you send it.
- Fix the output length with
max_tokensand a prompt that fills it, so tokens per second is comparable. - Report p50 and p95, not the mean. Tail latency is what users remember.
- Spread runs over time. Load changes during the day.
- Warm up first. The first call pays for DNS, TLS and connection setup; reuse one client.
What actually improves each number?
| To improve | Do this | Why |
|---|---|---|
| TTFT | Shorten or cache the prompt prefix | Prefill time grows with prompt length; cached prefixes skip work |
| TTFT | Keep connections warm, use a nearby region | Removes setup and round trips |
| Time to first content token | Lower reasoning effort, or use a non-reasoning model for simple steps | Less hidden thinking before the answer |
| Tokens per second | Use a smaller or faster model for the step | Decode speed depends on model size and serving |
| End-to-end | Ask for shorter output (schemas, word limits) | Output length multiplies ITL |
| End-to-end for many calls | Run independent calls in parallel within your concurrency limit | Wall-clock time drops even though each call is the same |
| Perceived latency | Stream to the user | They read while the model writes |
Output length is the most underrated lever. Asking for “JSON with these four fields” instead of “explain your answer” can cut generation time by an order of magnitude with no infrastructure work. Structured output helps here.
The cache lever is covered in prompt structure for caching. The parallelism lever runs into rate limits quickly; see handling 429 errors.
How does prompt length change the trade-off?
TTFT grows with the prompt because the model must process every input token before it can write. That is the hidden cost of very long prompts and part of the long context vs RAG decision: a 1M-token prompt answers questions RAG cannot, but it starts slower. If interactive speed matters, keep the hot path short and send the big context only when the question needs it.
Where BroadNet Inference fits
BroadNet publishes a typical speed of about 100 tokens per second for its plan model and streams with the standard SSE format, so the code above works unchanged. It does not publish TTFT figures or latency SLAs, so measure on your own prompts before you commit; a $1 Meteor hour is enough for a 50-run benchmark, and the starting credit covers one on Gemma 4 VL 26B. If you need guaranteed sub-second latency at large scale, test carefully: a pay-as-you-go key allows 4 requests in flight, which suits apps and agents more than high-volume real-time serving. Models and their prices are on the models and pricing pages.
