BroadRouter

How to handle LLM API 429 errors: backoff, jitter and limiters

By the BroadNet Inference team · 5 min read · Published · Updated

A 429 means you hit a rate, concurrency or spend limit. Retry it with capped exponential backoff and random jitter, and honour Retry-After when sent. Then prevent most 429s: cap requests in flight with a semaphore at the key's limit and queue the rest. Retry 5xx the same way; fix other 4xx errors instead.

A 429 means your traffic crossed a rate limit: requests per minute, requests in flight, tokens per minute or spend. Fix it in two halves. Retry well: capped exponential backoff with random jitter, honouring Retry-After. Send less at once: a semaphore at the key’s in-flight limit, plus a queue. Both are below, with code.

For a full status-code reference, see the errors and retries guide; for multi-hour agent runs with checkpoints and timeouts, see long-running agents: timeouts and retries.

What is the difference between 429 and 5xx?

Status Whose problem Retry? How
429 Your traffic crossed a limit Yes Backoff with jitter; honour Retry-After; lower concurrency
500 / 502 / 503 / 504 Server side Yes Bounded exponential backoff
Connection reset, timeout Network or server Usually Backoff; for streams, only if a new bill is acceptable
400 / 422 Bad request No Fix the payload
401 / 402 / 403 Key, balance or policy No Fix the key, fund the account, fix the allow-list
413 Request too large No Shrink it

On BroadNet Inference a 429 can mean any of four limits: rate, concurrency, token rate or spend. A spend limit will not clear in 30 seconds. BroadNet’s docs do not say which field of the error body names the limit, so log the body and compare it with the limits shown in your key details.

Our open dataset of LLM API free tiers and rate limits compares what 14 providers publish about free access, starting rate limits, card requirements and training on free-tier data, with the source and date for each value.

What limits cause 429s?

Know the numbers you are working against. On BroadNet Inference:

Key type Requests / minute In flight Other
Pay-as-you-go 30 4 Shared spend budget
Hourly key 120 1 stream 300,000 tokens / minute
Unlimited plan session — 1 request per session No token meter

The in-flight number matters most for LLMs. A streamed answer holds its slot until the last token arrives, so a long answer keeps it busy for as long as the model takes to write it (BroadNet gives a typical speed of about 100 tokens per second). Send more parallel calls than the key’s 4 slots and the extra ones get 429, even though you sent only a handful of requests that minute. That is a concurrency problem, and backoff alone handles it badly. To size workers against a limit before you launch, use the free LLM rate limit calculator. For a fixed job of many items, the LLM batch job planner turns the same limits into a run time. Other providers count differently (tokens a minute, requests a day or requests in flight); LLM API rate limits by provider compares 13 of them from their own docs.

How do you write backoff with jitter?

Plain exponential backoff waits 1, 2, 4, 8 seconds. If a hundred workers were throttled at the same moment, they all come back at the same moment. AWS’s simulation of 100 contending clients found that full jitter cut the number of calls by more than half and finished sooner than unjittered backoff. The simplest version, full jitter, waits a random time between zero and the exponential cap:

import os, random, time
from openai import OpenAI, RateLimitError, APIStatusError, APIConnectionError

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"], max_retries=0)

RETRYABLE = {429, 500, 502, 503, 504}

def call_with_backoff(max_attempts=6, base=1.0, cap=60.0, **kwargs):
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(**kwargs)
        except (RateLimitError, APIStatusError, APIConnectionError) as e:
            status = getattr(e, "status_code", None)
            if status is not None and status not in RETRYABLE:
                raise  # 400, 401, 403...: retrying will not help
            if attempt == max_attempts - 1:
                raise
            retry_after = None
            if getattr(e, "response", None) is not None:
                retry_after = e.response.headers.get("retry-after")
            wait = random.uniform(0, min(cap, base * 2 ** attempt))
            if retry_after and retry_after.isdigit():
                wait = max(wait, float(retry_after))
            time.sleep(wait)

max_retries=0 turns off the SDK’s own retries so you do not retry twice. If you are happy with the SDK’s built-in behaviour, the opposite works too: leave your loop out and set max_retries=4 on the client, as the Python SDK guide shows. The official Python and Node SDKs retry connection errors, 408, 409, 429 and 5xx twice by default.

Retry-After is a standard header (RFC 9110) that may hold seconds or a date. BroadNet’s docs do not promise it, so the code treats it as optional.

How do you stop 429s before they happen?

Cap requests in flight at the key’s limit. In Python, an asyncio.Semaphore does it with one object and one async with block. The code uses 4 as an example; set it to your key’s in-flight limit (4 on a BroadNet pay-as-you-go key):

import asyncio, os
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"], max_retries=4)
slots = asyncio.Semaphore(4)  # match the key's in-flight limit

async def ask(prompt):
    async with slots:
        r = await client.chat.completions.create(
            model="broadnet/gemma-4-26b-a4b",
            messages=[{"role": "user", "content": prompt}],
        )
        return r.choices[0].message.content

async def main(prompts):
    return await asyncio.gather(*(ask(p) for p in prompts))

results = asyncio.run(main(["Summarise A", "Summarise B", "Summarise C"]))

gather starts every task, but only as many run at once as the semaphore allows. The rest wait in the semaphore’s queue. In Node, the p-limit package does the same, again with 4 as the example value:

// ES module (top-level await). npm install openai p-limit
import OpenAI from "openai";
import pLimit from "p-limit";

const client = new OpenAI({ baseURL: "https://inference.broadnet.ai/v1", apiKey: process.env.BROADNET_API_KEY, maxRetries: 4 });
const limit = pLimit(4); // match the key's in-flight limit

async function ask(prompt) {
  const r = await client.chat.completions.create({
    model: "broadnet/gemma-4-26b-a4b",
    messages: [{ role: "user", content: prompt }],
  });
  return r.choices[0].message.content;
}

const prompts = ["Summarise A", "Summarise B", "Summarise C"];
const results = await Promise.all(prompts.map((p) => limit(() => ask(p))));

For a per-minute cap as well, add a token bucket that releases one permit every 60 / rpm seconds, where rpm is your key’s requests-a-minute limit (30 on a BroadNet pay-as-you-go key).

When do you need a real queue?

A semaphore lives in one process. Once several workers or servers share one key, the limit is shared and each process only sees its own traffic. Then:

  1. Put jobs on a shared queue (Redis, SQS, a database table).
  2. Run a fixed number of consumers equal to the key’s in-flight limit.
  3. Give each job an ID and store results by ID, so a retried job does not run twice.
  4. Send interactive traffic and batch traffic through separate keys or separate queues, so a batch backlog never blocks a user waiting on a reply.

For a bulk job of thousands of items with no Batch API, see the resumable batch runner.

Which retry mistakes turn 429s into outages?

  • Unbounded retries. A retry loop with no cap during a long outage multiplies your load and, on per-token billing, your bill.
  • Retrying a half-finished stream blindly. If the stream already returned tokens, a retry is a fresh inference and the user may see text twice. See the streaming post for detecting incomplete streams.
  • Retrying account errors. A 401 for an exhausted pay-as-you-go allowance or a 402 for insufficient balance will not fix itself.

Log every retry with its status, wait time and the X-Request-ID response header, which BroadNet’s docs ask you to keep for support. A rising retry rate is the earliest warning that you need more capacity; usage logging shows how to record it.

Where BroadNet Inference fits

BroadNet’s limits are published and small enough to design against: 4 requests in flight on a pay-as-you-go key. That suits apps and agents, and it is too tight for a bulk job that wants hundreds of parallel calls on one key. For agents, an unlimited plan has no token meter and runs one request at a time per session; you add sessions for parallel work, as described in running parallel coding agents. Current limits are in the API reference.

Questions

What does HTTP 429 mean on an LLM API?

Too Many Requests. You crossed a limit on requests per minute, requests in flight, tokens per minute or spend. The request was not processed, so it is safe to send again after a wait.

How long should I wait before retrying a 429?

Use Retry-After if the server sends it. Otherwise start near one second, double each attempt, cap around 30 to 60 seconds, and pick a random wait between zero and that cap. Stop after four to six attempts.

Should I retry 500 and 503 errors from an LLM API?

Yes, with the same bounded backoff. They are server-side. A 400, 401, 403 or 422 will fail the same way every time, so fix it instead.

Does the OpenAI SDK retry 429 automatically?

Yes. The official Python and Node SDKs retry connection errors, 408, 409, 429 and 5xx twice by default with a short exponential backoff. Raise max_retries for batch jobs and add your own concurrency limit.

Why do I get 429 when I am under the requests-per-minute limit?

Usually the concurrency limit. Long streaming responses stay in flight for many seconds, so a handful of parallel calls can hit the in-flight cap while the per-minute count looks low.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity