Every shared API protects itself with limits. Knowing which ones apply tells you how fast you can go and what to do when you hit one.
How it works
LLM APIs measure several things at once. OpenAI, for example, lists:
- RPM: requests per minute
- RPD: requests per day
- TPM: tokens per minute
- TPD: tokens per day
Whichever limit you hit first applies. Many providers also cap concurrency, the number of requests in flight.
Over the limit, the server returns 429 Too Many Requests. MDN describes it as the client having “sent too many requests in a given amount of time.” A Retry-After header may say how many seconds to wait.
A safe retry loop:
import random, time
for attempt in range(5):
r = send()
if r.status_code != 429:
break
wait = float(r.headers.get("Retry-After", 2 ** attempt))
time.sleep(wait + random.random())
The official OpenAI SDKs already retry 429 and 5xx errors with backoff; set max_retries instead of writing your own loop.
Why it matters
- Throughput. Your real ceiling is the lowest of all limits. On a key limited to 30 requests a minute, as BroadNet’s pay-as-you-go keys are, 1,000 requests take at least 1,000 ÷ 30 minutes, no matter how fast the model is. The rate limit calculator does the math for your own limits.
- Reliability. Unhandled 429s look like outages. Backoff turns them into short delays.
- Cost. Some providers return 429 for spending caps too, common on pay-as-you-go keys. Read the error body before retrying.
On BroadNet Inference
| Key type | Limits |
|---|---|
| Pay-as-you-go | 30 requests a minute, 4 in flight at once |
| Hourly key | 120 requests a minute, one stream, 300,000 tokens a minute |
| Unlimited plan | One request at a time per session, no token meter |
BroadNet returns 429 when a rate, concurrency, token-rate or spending limit is reached. Back off and retry. The errors and retries guide covers every status code, and what “unlimited” really means explains which limits a flat plan still has.
Common mistakes
- Retrying 401 or 403 errors. Those are key or policy problems, not rate limits.
- Running many workers on one key without a shared limiter. Each worker backs off alone while the others keep the key saturated.