---
title: "promptfoo openai:chat provider with a custom apiBaseUrl"
description: "Run promptfoo evals and side-by-side model tests on BroadNet Inference with the openai:chat provider, apiBaseUrl and apiKeyEnvar, plus graders."
url: https://broadrouter.com/guides/promptfoo-openai-compatible/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Evaluate BroadNet models with promptfoo

Updated 2026-11-16

In promptfooconfig.yaml, add a provider with id openai:chat:broadnet/glm-5.3 and a config that sets apiBaseUrl to https://inference.broadnet.ai/v1 and apiKeyEnvar to BROADNET\_API\_KEY. Use the openai:chat prefix so promptfoo always calls Chat Completions, the endpoint BroadNet serves. Set max\_tokens yourself, since promptfoo's default output limit is 1,024 tokens.

promptfoo is an open-source CLI for testing prompts, comparing models and red-teaming LLM apps. Its OpenAI provider works with OpenAI-compatible gateways through `apiBaseUrl`. BroadNet Inference serves chat completions at `https://inference.broadnet.ai/v1`, so you can compare BroadNet’s models with each other, or with models you already use, from one YAML file. BroadRouter is BroadNet’s own site; this follows promptfoo’s OpenAI provider docs as of release 0.125.1.

## Set up

```bash
npm install -g promptfoo     # or: npx promptfoo@latest
export BROADNET_API_KEY='your-broadnet-key'
```

## promptfooconfig.yaml

```yaml
description: BroadNet model comparison
prompts:
  - 'What is the capital of Saudi Arabia? Answer in one sentence.'
providers:
  - id: openai:chat:broadnet/glm-5.3
    label: GLM 5.3
    config:
      apiBaseUrl: https://inference.broadnet.ai/v1
      apiKeyEnvar: BROADNET_API_KEY
      max_tokens: 4000
  - id: openai:chat:broadnet/deepseek-v4.1-flash
    label: DeepSeek V4.1 Flash
    config:
      apiBaseUrl: https://inference.broadnet.ai/v1
      apiKeyEnvar: BROADNET_API_KEY
      omitDefaults: true
tests:
  - assert:
      - type: icontains
        value: Riyadh
```

Real configs usually put test inputs in `tests.vars` and reference them in the prompt with promptfoo’s double-brace template syntax. We kept this one fixed so it runs as written.

Run it:

```bash
promptfoo eval
promptfoo view   # side-by-side results in the browser
```

## What each setting does

-   **`openai:chat:` plus the BroadNet ID.** Everything after `openai:chat:` is the model ID sent, so keep the `broadnet/` prefix. The explicit `chat` part pins the Chat Completions endpoint; BroadNet doesn’t serve the Responses API.
-   **`apiBaseUrl`** is the base URL with `/v1`, but without `/chat/completions`. promptfoo adds the path.
-   **`apiKeyEnvar`** names the variable that holds your BroadNet key. promptfoo does not fall back to `OPENAI_API_KEY` if it’s missing.
-   **`max_tokens`**. promptfoo’s defaults for non-reasoning requests are `temperature: 0` and a 1,024-token limit. BroadNet’s DeepSeek, GLM and Qwen models reason before they answer, so 1,024 can produce empty answers. Set a higher limit, as in the first provider.
-   **`omitDefaults: true`** drops promptfoo’s default temperature and limit, as in the second provider. The request then has no output limit from promptfoo.

## Model-graded assertions

Provider settings don’t apply to graders such as `llm-rubric`. To grade with a BroadNet model, add it again for the grader:

```yaml
defaultTest:
  options:
    provider:
      id: openai:chat:broadnet/glm-5.3
      config:
        apiBaseUrl: https://inference.broadnet.ai/v1
        apiKeyEnvar: BROADNET_API_KEY
        max_tokens: 4000
```

## Which models to compare

| Model | Good for |
| --- | --- |
| `broadnet/glm-5.3` | Tool-calling and structured-output tests |
| `broadnet/qwen3.8-27b` | Tests with images, Arabic text or tools |
| `broadnet/deepseek-v4.1-flash` | Long-context and cost-sensitive tests |
| `broadnet/gemma-4-26b-a4b` | Free tests on the $5 starting credit |

BroadNet’s docs list tools and structured output for GLM 5.3 and Qwen3.8 27B only. [Prices for every model](https://broadrouter.com/pricing/).

## How we tested this

On 2026-11-15 we installed promptfoo 0.125.1 with Node 24 and ran `promptfoo eval` on a config like the one above, pointed at a local mock server that speaks BroadNet’s chat-completions shape. The tests passed. The logged requests went to `/v1/chat/completions` with the BroadNet model ID unchanged and a Bearer key. The first provider sent `max_tokens: 4000` and `temperature: 0`; the provider with `omitDefaults: true` sent only the model and messages. With `BROADNET_API_KEY` unset, promptfoo stopped with “Missing BROADNET\_API\_KEY”. We did not test graders, and we had no BroadNet key, so we did not test real model output.

## Troubleshooting

-   **Empty or cut-off outputs.** Raise `max_tokens`. [Why answers come back empty](https://broadrouter.com/guides/reasoning-tokens-empty-response/).
-   **404.** Check that `apiBaseUrl` ends in `/v1` and the model ID after `openai:chat:` keeps the `broadnet/` prefix. Paid models need a top-up; the $5 credit covers only `broadnet/gemma-4-26b-a4b` and the two Transcribe models.
-   **429 during large evals.** promptfoo ran our eval with a concurrency of 4. A pay-as-you-go key allows 30 requests a minute and 4 in flight, so lower it with `promptfoo eval -j 1` for big test sets. See [errors and retries](https://broadrouter.com/guides/errors-and-retries/).

## Terms in this guide

[Chat Completions API](https://broadrouter.com/glossary/chat-completions-api/)

The Chat Completions API is OpenAI's HTTP endpoint, POST /v1/chat/completions, that takes a model name and a list of role-tagged messages and returns the model's next message. Its request and response shape became the common format that OpenAI-compatible providers and SDKs reuse.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[Base URL](https://broadrouter.com/glossary/base-url/)

A base URL is the fixed root address of an API, such as https://api.example.com/v1, that a client puts in front of every endpoint path, such as /chat/completions. In OpenAI-compatible SDKs, changing the base URL, together with the API key and model ID, points the same code at a different provider.

[Eval (LLM evaluation)](https://broadrouter.com/glossary/eval/)

An eval (short for evaluation) is a repeatable test of a language model or prompt: a fixed set of inputs, a description of what a good output looks like, and a grader (code, a human or another model) that scores each output so that versions can be compared.

[Prepaid credit](https://broadrouter.com/glossary/prepaid-credit/)

Prepaid credit is money added to an API account before use. Each request deducts its cost from the balance, and when the balance runs out, metered requests stop until you top up. It caps spend at what you have paid in.

[Rate limit](https://broadrouter.com/glossary/rate-limit/)

A rate limit is a cap an API places on how much a client can use it in a time window, such as requests per minute or tokens per minute. Requests over the limit are refused, usually with HTTP 429 Too Many Requests, until usage falls back under it.

## Questions

### Should I use openai:chat or openai as the promptfoo provider prefix?

openai:chat. With openai:chat:broadnet/glm-5.3, promptfoo calls /v1/chat/completions, which is what BroadNet serves. promptfoo's docs say an unrecognised model name after a bare openai prefix also falls back to Chat Completions, but the explicit chat prefix doesn't depend on that rule.

### Why do BroadNet answers get cut off in promptfoo?

promptfoo sets a 1,024-token output limit by default for non-reasoning requests, and BroadNet's DeepSeek, GLM and Qwen models spend part of the output budget on reasoning. Set max\_tokens in the provider config, or set omitDefaults to true.

### Does apiKeyEnvar fall back to OPENAI\_API\_KEY?

No. promptfoo's docs say that if the named variable is missing it does not fall back to OPENAI\_API\_KEY. In our test it stopped with "Missing BROADNET\_API\_KEY".

### Can a BroadNet model grade llm-rubric assertions?

Yes, but promptfoo's docs say provider settings don't apply to graders. Set the grader separately under defaultTest.options.provider with the same id and config.

## Related guides

-   [First request to BroadNet Inference with curl](https://broadrouter.com/guides/curl-quickstart/)
-   [Use the OpenAI Node.js SDK with BroadNet Inference](https://broadrouter.com/guides/openai-node-sdk/)
-   [Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)
-   [BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/)
-   [All tools that work with BroadNet Inference](https://broadrouter.com/integrations/): SDKs, coding agents, chat UIs and automation tools, with exact settings for each

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=guides-promptfoo-openai-compatible&utm_content=page-cta) ·

**Sources**

-   [promptfoo docs, OpenAI provider (OpenAI-compatible gateways) (checked 2026-11-15)](https://www.promptfoo.dev/docs/providers/openai/)
-   [promptfoo on GitHub, release 0.125.1, 2026-10-25 (checked 2026-11-15)](https://github.com/promptfoo/promptfoo/releases/tag/0.125.1)
-   [BroadNet Inference API reference (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=guides-promptfoo-openai-compatible&utm_content=sources)
