BroadRouter
Get started

Rate limit

Updated

A rate limit is a cap an API places on how much a client can use it in a time window, such as requests per minute or tokens per minute. Requests over the limit are refused, usually with HTTP 429 Too Many Requests, until usage falls back under it.

Also called: RPM, TPM, requests per minute, tokens per minute, HTTP 429, throttling

Every shared API protects itself with limits. Knowing which ones apply tells you how fast you can go and what to do when you hit one.

How it works

LLM APIs measure several things at once. OpenAI, for example, lists:

  • RPM: requests per minute
  • RPD: requests per day
  • TPM: tokens per minute
  • TPD: tokens per day

Whichever limit you hit first applies. Many providers also cap concurrency, the number of requests in flight.

Over the limit, the server returns 429 Too Many Requests. MDN describes it as the client having “sent too many requests in a given amount of time.” A Retry-After header may say how many seconds to wait.

A safe retry loop:

import random, time

for attempt in range(5):
    r = send()
    if r.status_code != 429:
        break
    wait = float(r.headers.get("Retry-After", 2 ** attempt))
    time.sleep(wait + random.random())

The official OpenAI SDKs already retry 429 and 5xx errors with backoff; set max_retries instead of writing your own loop.

Why it matters

  • Throughput. Your real ceiling is the lowest of all limits. On a key limited to 30 requests a minute, as BroadNet’s pay-as-you-go keys are, 1,000 requests take at least 1,000 ÷ 30 minutes, no matter how fast the model is. The rate limit calculator does the math for your own limits.
  • Reliability. Unhandled 429s look like outages. Backoff turns them into short delays.
  • Cost. Some providers return 429 for spending caps too, common on pay-as-you-go keys. Read the error body before retrying.

On BroadNet Inference

Key type Limits
Pay-as-you-go 30 requests a minute, 4 in flight at once
Hourly key 120 requests a minute, one stream, 300,000 tokens a minute
Unlimited plan One request at a time per session, no token meter

BroadNet returns 429 when a rate, concurrency, token-rate or spending limit is reached. Back off and retry. The errors and retries guide covers every status code, and what “unlimited” really means explains which limits a flat plan still has.

Common mistakes

  • Retrying 401 or 403 errors. Those are key or policy problems, not rate limits.
  • Running many workers on one key without a shared limiter. Each worker backs off alone while the others keep the key saturated.

Questions

Should I retry a 429?

Yes, after a wait. Follow the Retry-After header if present; otherwise use exponential backoff with random jitter, and cap the number of attempts. Do not retry spending-limit or billing errors that share the status code.

Do failed requests count toward the limit?

Often, yes. OpenAI's guide says failed requests still count toward per-minute limits, so retrying instantly in a loop makes things worse.

What is the difference between a rate limit and a concurrency limit?

A rate limit counts requests or tokens over time. A concurrency limit counts requests running at the same moment. A key can hit either one first.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity