LLM rate limit and concurrency calculator
Updated
In short: Real throughput is the smallest of three ceilings: your RPM limit, TPM ÷ tokens per request, and concurrency × 60 ÷ latency in seconds. Little's law gives the requests you need in flight: requests per second × latency. For example, 20 in flight with 6-second responses allows 200 requests a minute. Enter your limits to see which binds first.
A steady-state estimate: it assumes evenly spread traffic and the average latency you enter. Real traffic bursts, latency varies, and some providers count max_tokens rather than actual output against token limits, so keep 20–30% headroom and retry 429s with backoff.
Report a wrong price or number (email to info@broadnet.ai, with this page's link and your inputs filled in).
How it works
- Pick a preset or enter your provider's limits: requests per minute, tokens per minute (combined, or separate input and output limits) and maximum requests in flight. Leave a box empty if that limit does not apply.
- Enter average input and output tokens per request and the average latency in seconds.
- Read the sustainable requests per minute under each limit, which one binds, and how many requests you need in flight (Little's law).
- Enter a batch size to see how long it takes to process. Copy the link to share the result.
The formulas
max RPM under RPM limit = RPM
max RPM under TPM limit = TPM / (input + output tokens per request)
(separate limits) = min(input TPM / input, output TPM / output)
max RPM under concurrency limit = concurrency × 60 / latency_seconds
sustainable RPM = the smallest of the above
in flight needed (Little's law) = RPM / 60 × latency_seconds
time for N requests = N / sustainable RPM minutesLittle's law (L = λW) holds for any stable queue: the average number of requests in flight equals the arrival rate times how long each one takes. It is why slow reasoning models hit concurrency limits long before they hit request limits. See rate limit and tokens per second.
BroadNet Inference limits
| Access | Requests / min | In flight | Tokens / min |
|---|---|---|---|
| Pay-as-you-go key | 30 | 4 | Not published |
| Plan session (DeepSeek V4.1 Flash) | No meter | 1 per session | No meter |
| Hourly key (BroadNet Gemma 4 VL 26B) | 120 | 1 | 300,000 |
To compare BroadNet with other providers, see our open dataset of LLM API free tiers and rate limits, which records what 14 providers publish, with sources.
Worked example
A pay-as-you-go key allows 30 requests a minute and 4 in flight. With 6-second responses, concurrency allows 4 × 60 ÷ 6 = 40 requests a minute, so the 30 RPM limit binds. To reach 30 a minute you would need 30 ÷ 60 × 6 = 3 requests in flight. A batch of 10,000 requests then takes about 6 hours. A plan session at the same latency does 10 requests a minute, and two sessions do 20.
When you hit the limit
- Cap your client's concurrency at the provider's limit instead of letting requests queue and time out.
- Retry 429 and 5xx responses with exponential backoff and jitter; fix 4xx errors first. Error codes and retries.
- Cut latency: lower
max_tokens, setreasoning_effortto low, or stream. - Spread work across keys or plan sessions, within your provider's terms.
Questions
How do I calculate LLM API throughput from rate limits?
Work out the requests per minute each limit allows and take the smallest. RPM allows RPM. TPM allows TPM ÷ tokens per request. A concurrency limit allows concurrency × 60 ÷ latency in seconds. The smallest of the three is your real ceiling, and that limit is the one that binds.
What is Little's law and how does it apply to LLM APIs?
Little's law says the average number of requests in flight equals arrival rate × time in system (L = λW). For an API: requests needed in flight = requests per second × average latency. At 2 requests a second and 8-second responses you need 16 in flight; with a concurrency cap of 4 you top out at 0.5 a second.
Which limit binds first: RPM, TPM or concurrency?
For short prompts with fast answers, RPM usually binds. For long prompts or long answers, TPM binds. For reasoning models and long outputs, latency is high, so concurrency binds. Enter your numbers and the calculator names the binding limit.
Why do I get 429 errors below my RPM limit?
Another limit is binding: tokens per minute, requests in flight, or a short-window burst limit. Some providers also count max_tokens, not actual output, against token limits. Retry 429s with exponential backoff and jitter. Error codes and retries.
What are the BroadNet Inference rate limits?
A pay-as-you-go key allows 30 requests a minute and 4 requests in flight. A plan session runs one request at a time with no token meter. The $0.50 hourly key allows 120 requests a minute, 1 stream and 300,000 tokens a minute.
How do I estimate latency?
Latency ≈ time to first token + output tokens ÷ output speed. At about 100 tokens a second, 2,000 output tokens (including reasoning) take about 20 seconds plus queueing. Log real response times; p50 is the right input here, and p95 tells you how much headroom to keep.