promptfoo is an open-source CLI for testing prompts, comparing models and red-teaming LLM apps. Its OpenAI provider works with OpenAI-compatible gateways through apiBaseUrl. BroadNet Inference serves chat completions at https://inference.broadnet.ai/v1, so you can compare BroadNet’s models with each other, or with models you already use, from one YAML file. BroadRouter is BroadNet’s own site; this follows promptfoo’s OpenAI provider docs as of release 0.125.1.
Set up
npm install -g promptfoo # or: npx promptfoo@latest
export BROADNET_API_KEY='your-broadnet-key'
promptfooconfig.yaml
description: BroadNet model comparison
prompts:
- 'What is the capital of Saudi Arabia? Answer in one sentence.'
providers:
- id: openai:chat:broadnet/glm-5.3
label: GLM 5.3
config:
apiBaseUrl: https://inference.broadnet.ai/v1
apiKeyEnvar: BROADNET_API_KEY
max_tokens: 4000
- id: openai:chat:broadnet/deepseek-v4.1-flash
label: DeepSeek V4.1 Flash
config:
apiBaseUrl: https://inference.broadnet.ai/v1
apiKeyEnvar: BROADNET_API_KEY
omitDefaults: true
tests:
- assert:
- type: icontains
value: Riyadh
Real configs usually put test inputs in tests.vars and reference them in the prompt with promptfoo’s double-brace template syntax. We kept this one fixed so it runs as written.
Run it:
promptfoo eval
promptfoo view # side-by-side results in the browser
What each setting does
openai:chat:plus the BroadNet ID. Everything afteropenai:chat:is the model ID sent, so keep thebroadnet/prefix. The explicitchatpart pins the Chat Completions endpoint; BroadNet doesn’t serve the Responses API.apiBaseUrlis the base URL with/v1, but without/chat/completions. promptfoo adds the path.apiKeyEnvarnames the variable that holds your BroadNet key. promptfoo does not fall back toOPENAI_API_KEYif it’s missing.max_tokens. promptfoo’s defaults for non-reasoning requests aretemperature: 0and a 1,024-token limit. BroadNet’s DeepSeek, GLM and Qwen models reason before they answer, so 1,024 can produce empty answers. Set a higher limit, as in the first provider.omitDefaults: truedrops promptfoo’s default temperature and limit, as in the second provider. The request then has no output limit from promptfoo.
Model-graded assertions
Provider settings don’t apply to graders such as llm-rubric. To grade with a BroadNet model, add it again for the grader:
defaultTest:
options:
provider:
id: openai:chat:broadnet/glm-5.3
config:
apiBaseUrl: https://inference.broadnet.ai/v1
apiKeyEnvar: BROADNET_API_KEY
max_tokens: 4000
Which models to compare
| Model | Good for |
|---|---|
broadnet/glm-5.3 |
Tool-calling and structured-output tests |
broadnet/qwen3.8-27b |
Tests with images, Arabic text or tools |
broadnet/deepseek-v4.1-flash |
Long-context and cost-sensitive tests |
broadnet/gemma-4-26b-a4b |
Free tests on the $5 starting credit |
BroadNet’s docs list tools and structured output for GLM 5.3 and Qwen3.8 27B only. Prices for every model.
How we tested this
On 2026-11-15 we installed promptfoo 0.125.1 with Node 24 and ran promptfoo eval on a config like the one above, pointed at a local mock server that speaks BroadNet’s chat-completions shape. The tests passed. The logged requests went to /v1/chat/completions with the BroadNet model ID unchanged and a Bearer key. The first provider sent max_tokens: 4000 and temperature: 0; the provider with omitDefaults: true sent only the model and messages. With BROADNET_API_KEY unset, promptfoo stopped with “Missing BROADNET_API_KEY”. We did not test graders, and we had no BroadNet key, so we did not test real model output.
Troubleshooting
- Empty or cut-off outputs. Raise
max_tokens. Why answers come back empty. - 404. Check that
apiBaseUrlends in/v1and the model ID afteropenai:chat:keeps thebroadnet/prefix. Paid models need a top-up; the $5 credit covers onlybroadnet/gemma-4-26b-a4band the two Transcribe models. - 429 during large evals. promptfoo ran our eval with a concurrency of 4. A pay-as-you-go key allows 30 requests a minute and 4 in flight, so lower it with
promptfoo eval -j 1for big test sets. See errors and retries.
