BroadRouter

Long-running agents: timeouts, retries and checkpoints that hold

By the BroadNet Inference team · 4 min read · Published · Updated

Treat each model call as disposable and the agent's state as durable. Set a timeout per call, stream to detect stalls, retry 429, 5xx and dropped streams with bounded backoff, and give every side-effecting tool an idempotency key. Save the message list after each step so a crash resumes from the last checkpoint, not from the start.

An agent that runs for five minutes can ignore failure. One that runs for five hours will see a timeout, a 429, a dropped stream and probably a restart. Most guides on long-running agents sell a workflow engine. You may not need one. This post covers the five habits that keep a plain agent loop alive for hours, with code for the OpenAI Python SDK.

What breaks when an agent runs for hours?

Failure How it looks Habit that handles it
Hung or slow call No reply for minutes Per-call timeout, streaming
Rate limit or overload 429, 5xx Bounded retry with backoff
Dropped stream Connection ends before [DONE] Treat as incomplete, retry the call
Duplicate side effect Tool runs twice after a retry Idempotency keys
Process crash or deploy Agent restarts at step 1 Checkpoint per step

1. How long should a call be allowed to take?

The OpenAI Python SDK’s README says requests time out after 10 minutes by default and certain errors are retried twice. For an agent, set both yourself, so the behavior doesn’t change under you:

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"],
                timeout=300, max_retries=4)

Pick the timeout from data. Log the duration of every call for a day; set the timeout well above the slowest successful call. Large prompts take longer to start, and reasoning models think before writing. At about 100 tokens per second, 8,000 output tokens take over a minute on their own. Add a separate wall-clock budget per task so a loop that keeps going can’t run all night.

2. Why stream in a long-running agent?

Streaming gives you a signal while the model works. Without it, a slow call and a dead connection look the same until the timeout fires. With it, you can see chunks arrive and also get token usage in the last chunk.

BroadNet’s streaming rules: read until the literal data: [DONE] line. If the connection closes before it, the answer is incomplete. Record the X-Request-ID, then decide whether to make a new request. Full streaming guide.

def call_streaming(messages, tools=None):
    parts, done = [], False
    stream = client.chat.completions.create(
        model="broadnet/glm-5.3", messages=messages, tools=tools, stream=True,
        stream_options={"include_usage": True}, max_tokens=8000)
    for chunk in stream:
        if chunk.choices and chunk.choices[0].finish_reason:
            done = True
        if chunk.choices and chunk.choices[0].delta.content:
            parts.append(chunk.choices[0].delta.content)
    if not done:
        raise IncompleteStream()   # retry the whole call
    return "".join(parts)

This sketch collects text only. If you stream tool calls, accumulate delta.tool_calls by index as well.

3. What should you retry?

The SDK already retries connection errors, 408, 409, 429 and 5xx. Layer your own retry on top only for what it can’t see, such as an incomplete stream:

  1. Retry: connection errors, 429, 5xx, incomplete streams. Use exponential backoff with jitter and a cap, for example 4 attempts.
  2. Fix, don’t retry: 400, 401, 402, 403, 413, 422. These fail the same way every time. See error codes and when to retry.
  3. Budget retries per task, not only per call. Ten steps each retrying four times is forty extra calls.
  4. Fail over once retries are spent. If a run must survive a provider outage, move connection errors, 5xx and rate-limit 429s to a second provider. LLM API failover shows the client code and a circuit breaker.

On pay-as-you-go, every retry is a new inference, billed again. A pay-as-you-go key also allows 30 requests a minute and 4 in flight, so retry storms hit the rate limit quickly; the rate limit calculator shows how much headroom your agent has. The 429 handling post covers backoff in detail. On an unlimited plan, retries cost nothing extra, but they occupy the session’s single request slot while they run.

4. How do you make tool calls idempotent?

A model call has no side effects. A tool call can. If the agent crashes after send_invoice ran but before the step was saved, the restarted agent will send it again.

Give each side-effecting call an idempotency key derived from the task and step, check it before acting, and skip the call if it’s already recorded:

def run_side_effect(task_id, step, call, store):
    key = f"{task_id}:{step}:{call.id}"
    if store.done(key):
        return store.result(key)       # already ran: return the saved result
    result = execute(call)             # pass key to the API if it supports one
    store.save(key, result)
    return result

Many payment and messaging APIs accept an idempotency key header. Pass yours through when they do.

5. How do you checkpoint an agent?

Save after every completed step: the message list, the step number and the side-effect keys. On start, load the latest checkpoint if one exists.

def save_checkpoint(path, step, messages):
    tmp = path + ".tmp"
    with open(tmp, "w") as f:
        json.dump({"step": step, "messages": messages}, f)
    os.replace(tmp, path)   # atomic on the same filesystem

Writing to a temp file and renaming means a crash mid-write never leaves a half checkpoint. For very long sessions, also checkpoint a summary: replace old turns with a short summary so a resumed agent doesn’t reload a huge history. That keeps you inside the context window, even at 1,048,576 tokens on DeepSeek V4.1 Flash.

A checklist before an overnight run

  1. Timeout and max_retries set explicitly on the client.
  2. Streaming on, with include_usage, and incomplete streams retried.
  3. A retry budget per task and a wall-clock limit.
  4. Idempotency keys on every tool that writes, sends or pays.
  5. A checkpoint after every step, written atomically.
  6. Logs with step number, duration, tokens and X-Request-ID.

If you run several agents side by side, read running parallel coding agents on a flat rate and what multi-agent setups cost.

Where BroadNet Inference fits

BroadNet Inference uses standard chat-completions streaming, returns an X-Request-ID header you can log, and documents which errors to retry, so the patterns above work as written. For agents that run all day, the unlimited plans remove the token meter on DeepSeek V4.1 Flash for $60 a month per session, which makes retries free. Keep in mind that a Meteor hour ends one hour after purchase and an Orbit week after seven days, so a long job must finish inside the period or move to a renewing Full Moon session. Plans are at inference.broadnet.ai/plans.

Questions

What timeout should I set for LLM API calls in an agent?

Long enough for the slowest legitimate call, short enough to catch a hung one. The OpenAI Python SDK defaults to 10 minutes. Many agents use 2 to 5 minutes per call with streaming, and a separate wall-clock limit per task.

Which LLM API errors should an agent retry?

Retry connection errors, 429 and 5xx with bounded exponential backoff and jitter. Fix 400, 401, 402, 403, 413 and 422 instead of retrying. On BroadNet, also log the X-Request-ID header for each failure.

Is it safe to retry a chat completion?

The model call itself has no side effects, so yes, but it is a new inference and is billed again on pay-as-you-go. The danger is in tools. A retried step can send an email twice unless the tool uses an idempotency key.

How do I resume an agent after a crash?

Save the message list and step number after every completed step, write side-effect IDs before you act, and reload them on start. The agent resumes from the last saved step.

Do long-running agents need an unlimited plan?

No, but retries and long contexts add up on pay-as-you-go. On BroadNet's unlimited plans, DeepSeek V4.1 Flash has no token meter, so retries cost nothing extra, though each session still runs one request at a time.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity