BroadRouter
Get started

Evaluate BroadNet models with promptfoo

Updated

In promptfooconfig.yaml, add a provider with id openai:chat:broadnet/glm-5.3 and a config that sets apiBaseUrl to https://inference.broadnet.ai/v1 and apiKeyEnvar to BROADNET_API_KEY. Use the openai:chat prefix so promptfoo always calls Chat Completions, the endpoint BroadNet serves. Set max_tokens yourself, since promptfoo's default output limit is 1,024 tokens.

promptfoo is an open-source CLI for testing prompts, comparing models and red-teaming LLM apps. Its OpenAI provider works with OpenAI-compatible gateways through apiBaseUrl. BroadNet Inference serves chat completions at https://inference.broadnet.ai/v1, so you can compare BroadNet’s models with each other, or with models you already use, from one YAML file. BroadRouter is BroadNet’s own site; this follows promptfoo’s OpenAI provider docs as of release 0.125.1.

Set up

npm install -g promptfoo     # or: npx promptfoo@latest
export BROADNET_API_KEY='your-broadnet-key'

promptfooconfig.yaml

description: BroadNet model comparison
prompts:
  - 'What is the capital of Saudi Arabia? Answer in one sentence.'
providers:
  - id: openai:chat:broadnet/glm-5.3
    label: GLM 5.3
    config:
      apiBaseUrl: https://inference.broadnet.ai/v1
      apiKeyEnvar: BROADNET_API_KEY
      max_tokens: 4000
  - id: openai:chat:broadnet/deepseek-v4.1-flash
    label: DeepSeek V4.1 Flash
    config:
      apiBaseUrl: https://inference.broadnet.ai/v1
      apiKeyEnvar: BROADNET_API_KEY
      omitDefaults: true
tests:
  - assert:
      - type: icontains
        value: Riyadh

Real configs usually put test inputs in tests.vars and reference them in the prompt with promptfoo’s double-brace template syntax. We kept this one fixed so it runs as written.

Run it:

promptfoo eval
promptfoo view   # side-by-side results in the browser

What each setting does

  • openai:chat: plus the BroadNet ID. Everything after openai:chat: is the model ID sent, so keep the broadnet/ prefix. The explicit chat part pins the Chat Completions endpoint; BroadNet doesn’t serve the Responses API.
  • apiBaseUrl is the base URL with /v1, but without /chat/completions. promptfoo adds the path.
  • apiKeyEnvar names the variable that holds your BroadNet key. promptfoo does not fall back to OPENAI_API_KEY if it’s missing.
  • max_tokens. promptfoo’s defaults for non-reasoning requests are temperature: 0 and a 1,024-token limit. BroadNet’s DeepSeek, GLM and Qwen models reason before they answer, so 1,024 can produce empty answers. Set a higher limit, as in the first provider.
  • omitDefaults: true drops promptfoo’s default temperature and limit, as in the second provider. The request then has no output limit from promptfoo.

Model-graded assertions

Provider settings don’t apply to graders such as llm-rubric. To grade with a BroadNet model, add it again for the grader:

defaultTest:
  options:
    provider:
      id: openai:chat:broadnet/glm-5.3
      config:
        apiBaseUrl: https://inference.broadnet.ai/v1
        apiKeyEnvar: BROADNET_API_KEY
        max_tokens: 4000

Which models to compare

Model Good for
broadnet/glm-5.3 Tool-calling and structured-output tests
broadnet/qwen3.8-27b Tests with images, Arabic text or tools
broadnet/deepseek-v4.1-flash Long-context and cost-sensitive tests
broadnet/gemma-4-26b-a4b Free tests on the $5 starting credit

BroadNet’s docs list tools and structured output for GLM 5.3 and Qwen3.8 27B only. Prices for every model.

How we tested this

On 2026-11-15 we installed promptfoo 0.125.1 with Node 24 and ran promptfoo eval on a config like the one above, pointed at a local mock server that speaks BroadNet’s chat-completions shape. The tests passed. The logged requests went to /v1/chat/completions with the BroadNet model ID unchanged and a Bearer key. The first provider sent max_tokens: 4000 and temperature: 0; the provider with omitDefaults: true sent only the model and messages. With BROADNET_API_KEY unset, promptfoo stopped with “Missing BROADNET_API_KEY”. We did not test graders, and we had no BroadNet key, so we did not test real model output.

Troubleshooting

  • Empty or cut-off outputs. Raise max_tokens. Why answers come back empty.
  • 404. Check that apiBaseUrl ends in /v1 and the model ID after openai:chat: keeps the broadnet/ prefix. Paid models need a top-up; the $5 credit covers only broadnet/gemma-4-26b-a4b and the two Transcribe models.
  • 429 during large evals. promptfoo ran our eval with a concurrency of 4. A pay-as-you-go key allows 30 requests a minute and 4 in flight, so lower it with promptfoo eval -j 1 for big test sets. See errors and retries.

Questions

Should I use openai:chat or openai as the promptfoo provider prefix?

openai:chat. With openai:chat:broadnet/glm-5.3, promptfoo calls /v1/chat/completions, which is what BroadNet serves. promptfoo's docs say an unrecognised model name after a bare openai prefix also falls back to Chat Completions, but the explicit chat prefix doesn't depend on that rule.

Why do BroadNet answers get cut off in promptfoo?

promptfoo sets a 1,024-token output limit by default for non-reasoning requests, and BroadNet's DeepSeek, GLM and Qwen models spend part of the output budget on reasoning. Set max_tokens in the provider config, or set omitDefaults to true.

Does apiKeyEnvar fall back to OPENAI_API_KEY?

No. promptfoo's docs say that if the named variable is missing it does not fall back to OPENAI_API_KEY. In our test it stopped with "Missing BROADNET_API_KEY".

Can a BroadNet model grade llm-rubric assertions?

Yes, but promptfoo's docs say provider settings don't apply to graders. Set the grader separately under defaultTest.options.provider with the same id and config.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity