BroadRouter

Stream chat completions and handle dropped connections

Updated

Set stream to true on /v1/chat/completions and read Server-Sent Events until the literal data: [DONE] marker. Add stream_options include_usage to get token counts in the last chunk. If the connection ends before [DONE], treat the answer as incomplete and log the X-Request-ID.

Streaming lets an agent or a chat interface show the answer as it is generated. BroadNet Inference uses the standard chat-completions streaming shape, so the OpenAI Python SDK and OpenAI Node SDK handle the parsing for you.

With curl

curl -N -D stream-headers.txt https://inference.broadnet.ai/v1/chat/completions \
  -H "Authorization: Bearer $BROADNET_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"broadnet/gemma-4-26b-a4b","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Count from one to three."}]}'

-N turns off curl’s buffering, and -D saves the headers, including X-Request-ID.

What does the stream look like?

The response is a Server-Sent Events stream. Under the WHATWG HTML standard, each event is one or more data: lines followed by a blank line, and a line that starts with a colon is a comment you can ignore. Each data: line holds one JSON chunk, and the text arrives in choices[0].delta.content.

The rules

  1. Read events until the literal data: [DONE] line, then close the reader.
  2. With stream_options: {"include_usage": true}, one extra chunk arrives just before [DONE]. It carries usage for the whole request and an empty choices list, so check choices before you index into it.
  3. When the model reports them, that usage includes completion_tokens_details.reasoning_tokens (DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason first) and prompt_tokens_details.cached_tokens. Cached tokens are billed at the lower cached-input price.

With the Python SDK

import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

stream = client.chat.completions.create(
    model="broadnet/gemma-4-26b-a4b",
    messages=[{"role": "user", "content": "Count from one to three."}],
    stream=True,
    stream_options={"include_usage": True},
)
usage = None
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        usage = chunk.usage
print()
print(usage)  # None means the stream ended before the usage chunk

The SDK reads up to [DONE] for you. The OpenAI Node SDK guide shows the same loop in JavaScript.

When the stream ends early

If the connection closes before [DONE], the answer is incomplete. BroadNet’s guidance:

  • record the X-Request-ID;
  • decide whether your application can safely make a new request. A retry is a new inference and is billed again on pay-as-you-go.

For agents on an unlimited plan, retries cost nothing extra, but they still occupy the session while they run.

BroadNet’s docs say per-model release verification is still required before you rely on streaming in production, so test it on the model you ship with.

Troubleshooting

  • Everything prints at the end. curl needs -N. Behind your own proxy, turn off response buffering for text/event-stream.
  • IndexError or undefined on the last chunk. The usage chunk has an empty choices list. Check chunk.choices before reading [0].
  • No usage at all. You did not send stream_options, or the stream broke before the final chunk.
  • Stream stops with finish_reason length and little text. A reasoning model spent max_tokens on reasoning. See why reasoning models return empty answers.
  • 429 while streaming in parallel. Each open stream counts as a request in flight; a pay-as-you-go key allows 4. See BroadNet error codes and retries.

Questions

How do I get token usage when streaming chat completions?

Send stream_options with include_usage set to true. One extra chunk arrives just before data: [DONE], with usage filled in and an empty choices array. If the stream breaks first, that chunk never arrives.

Why does curl print the whole streamed answer at once?

curl buffers its output by default. Add the -N flag (no buffering) so each Server-Sent Event prints as it arrives.

Is a retried stream billed twice?

On pay-as-you-go, yes. A retry is a new inference request. On an unlimited DeepSeek V4.1 Flash plan there is no token meter, but the retry still occupies the session while it runs.

Does streaming cost more than a normal request?

No. Streaming changes how the answer is delivered, not how it is billed. On pay-as-you-go you pay the same per-token input and output prices as for a request without streaming.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity