---
title: "Evaluate an LLM before switching - eval set and LLM judge"
description: "Build a small eval set from real traffic, score it with code checks and a debiased LLM judge, and compare quality, cost and latency before you migrate."
url: https://broadrouter.com/blog/evaluate-llm-before-switching/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# How to evaluate an LLM before switching providers

By the BroadNet Inference team · 5 min read · Published 2026-11-14

Collect 50 to 200 real prompts with known good outcomes. Score both models with code checks first and an LLM judge only where code cannot decide. Swap answer order to cancel position bias, use a judge from a third model family, check it against your own labels, and fix the pass rule, including cost and latency, before you run it.

## Key takeaways

-   Build the eval set from real logs, not invented prompts, and include the hard and failing cases.
-   Prefer deterministic checks (JSON parses, field equals, test passes). Use an LLM judge only for qualities code cannot check.
-   LLM judges show position, verbosity and self-preference bias. Judge each pair twice with the order swapped.
-   With 100 cases, a score of 80% has a 95% margin of about ±8 points. Small differences are noise.
-   Decide the pass rule in advance, including cost, latency and no regressions on critical cases.

**On this page**

1.  How do you build an eval set from real traffic?
2.  Which checks should be code and which should be a judge?
3.  What goes wrong with LLM-as-a-judge?
4.  How do you run a side-by-side comparison?
5.  How big a difference is real?
6.  What should the decision rule be?
7.  Where BroadNet Inference fits

To evaluate an LLM before switching, run both models on 50 to 200 real prompts with known good outcomes. Score with code checks first and a debiased LLM judge only where code cannot decide, log cost and latency, and fix the pass rule in advance.

The code change itself is small: with an [OpenAI-compatible API](https://broadrouter.com/glossary/openai-compatible-api/) it is a [base URL](https://broadrouter.com/glossary/base-url/) and a model ID. The [migration checklist](https://broadrouter.com/blog/openai-compatible-api-migration-checklist/) covers that part, and the [provider checklist](https://broadrouter.com/blog/choose-llm-api-provider-checklist/) covers price, limits and privacy. The hard part is knowing whether the new model is as good **for your tasks**. Public leaderboards do not answer that; [Artificial Analysis and its alternatives](https://broadrouter.com/blog/artificial-analysis-alternatives/) are for building a shortlist. A small [eval](https://broadrouter.com/glossary/eval/) set built from your own traffic does, in a day or two of work.

## How do you build an eval set from real traffic?

1.  **Pull 300 to 1,000 recent requests** from your [usage logs](https://broadrouter.com/blog/llm-api-usage-logging/), after redaction.
2.  **Group by task type**: extraction, classification, summarisation, chat reply, code edit, tool call.
3.  **Sample 10 to 40 per type**, more for high-volume or high-risk types. Include long prompts, non-English inputs and the cases users complained about.
4.  **Write the expected outcome** for each: an exact value, a schema, a test that must pass, or a short rubric (“mentions the refund window; no invented policy”).
5.  **Tag critical cases.** A wrong invoice total or a leaked instruction is not the same as a slightly worse tone.
6.  **Freeze it** in version control. Add cases when you find new failures, never delete them to make a model look better.

## Which checks should be code and which should be a judge?

Use the cheapest reliable check for each case:

| Task | Check | Judge needed? |
| --- | --- | --- |
| JSON extraction | Parses, schema validates, fields equal expected | No |
| Classification | Label equals expected | No |
| Code edit | Tests pass | No |
| Tool call | Right function, arguments validate | No |
| Summary | Contains required facts; no facts absent from source | Sometimes |
| Chat reply, tone, helpfulness | Rubric | Yes |

Deterministic checks are free, repeatable and impossible to flatter, so push as many cases into code checks as you can.

## What goes wrong with LLM-as-a-judge?

An [LLM-as-a-judge](https://broadrouter.com/glossary/llm-as-judge/) is a model that grades other models’ answers against a rubric. The MT-Bench study (Zheng et al.) found that strong judges agreed with human preferences more than 80% of the time, about as often as humans agree with each other. The same paper named the biases you have to design around:

| Bias | What happens | Mitigation |
| --- | --- | --- |
| Position | Prefers the first (or second) answer shown | Judge every pair twice, order swapped; count a win only if both agree |
| Verbosity | Prefers longer answers | Rubric that rewards correctness and brevity; cap length in the prompt |
| Self-preference | Prefers answers in its own style | Judge from a different model family than both candidates |
| Weak reasoning | Misses errors in maths or code | Use code checks for those cases instead |

Two more rules from practice. **Pin the judge**: same model version, same prompt, temperature 0, for the whole comparison. And **calibrate it**: label 30 to 50 pairs yourself, run the judge on them, and only trust it if it agrees with you most of the time. Arize’s evaluator guide gives the same advice (“Collect 20–100 examples” with human labels) and recommends categorical labels such as pass/fail over large numeric scales.

## How do you run a side-by-side comparison?

Run both candidates on every case, blind the judge, swap order, and log cost and time:

```python
import os, random, time, json
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
A, B = "broadnet/deepseek-v4.1-flash", "broadnet/qwen3.8-27b"
JUDGE = "broadnet/glm-5.3"  # a third family, pinned

def run(model, case):
    t0 = time.perf_counter()
    r = client.chat.completions.create(model=model, messages=case["messages"], max_tokens=4000, temperature=0)
    return {"text": r.choices[0].message.content or "", "usage": r.usage.model_dump(),
            "secs": time.perf_counter() - t0}

def judge(case, first, second):
    prompt = (f"Task:\n{case['task']}\n\nRubric:\n{case['rubric']}\n\n"
              f"Answer 1:\n{first}\n\nAnswer 2:\n{second}\n\n"
              'Reply with JSON only: {"winner": 1 or 2 or 0 for tie, "reason": "..."}')
    r = client.chat.completions.create(model=JUDGE, temperature=0, max_tokens=2000,
                                       response_format={"type": "json_object"},
                                       messages=[{"role": "user", "content": prompt}])
    return int(json.loads(r.choices[0].message.content or "{}").get("winner", 0))

def compare(case):
    a, b = run(A, case), run(B, case)
    if case.get("check"):                       # deterministic first
        return {"a_ok": case["check"](a["text"]), "b_ok": case["check"](b["text"]), "a": a, "b": b}
    w1 = judge(case, a["text"], b["text"])      # A shown first
    w2 = judge(case, b["text"], a["text"])      # B shown first
    winner = "A" if (w1, w2) == (1, 2) else "B" if (w1, w2) == (2, 1) else "tie"
    return {"winner": winner, "a": a, "b": b}
```

The swap rule means a model wins only when the judge prefers it in both positions; any disagreement counts as a tie. The share of disagreements is itself useful: a high rate means the judge or the rubric is too vague. Note the `max_tokens=4000`: on reasoning models, thinking counts toward the limit, and a low cap produces empty answers that look like failures. See [why reasoning models return empty replies](https://broadrouter.com/guides/reasoning-tokens-empty-response/).

## How big a difference is real?

Eval scores are samples. For a pass rate p over n cases, the standard error is √(p × (1 − p) ÷ n), and a 95% margin is about twice that.

| Cases | Pass rate | 95% margin |
| --- | --- | --- |
| 50 | 80% | about ±11 points |
| 100 | 80% | about ±8 points |
| 200 | 80% | about ±6 points |
| 400 | 80% | about ±4 points |

So a model scoring 82% against 79% on 100 cases has not been shown to be better. Look at **per-case differences** instead: which cases flipped from pass to fail? Those tell you more than the totals, and on critical cases one flip can decide the switch.

## What should the decision rule be?

Write it before you run the eval, so the result cannot bend it. A practical template:

1.  No regression on any case tagged critical.
2.  Overall pass rate not lower than the current model’s by more than the margin of error.
3.  p95 latency within your budget; see [TTFT vs throughput](https://broadrouter.com/blog/llm-latency-ttft-throughput/) for how to measure it.
4.  Cost per successful case lower, or higher only by an amount you have agreed is worth the quality. Price the monthly difference with the [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/).
5.  A shadow period: send a share of live traffic to the new model, log both, and compare before you cut over.

The same eval set is the starting point if you are considering a custom model: [before fine-tuning](https://broadrouter.com/blog/fine-tuning-readiness/) covers when prompts and retrieval are enough.

## Where BroadNet Inference fits

BroadNet Inference serves several model families behind one OpenAI-compatible endpoint, including [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/), [GLM 5.3](https://broadrouter.com/models/glm-5.3/) and [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/), so candidates and a judge from a different family can run through the same client and key. Every response carries token usage for cost comparison, and new accounts start with $5 of credit; without a top-up that credit runs only [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/), so plan a small top-up to compare the larger models. It does not include an evaluation harness or hosted judge tooling; you run the code above yourself. For Arabic workloads, [choosing an LLM for Arabic](https://broadrouter.com/blog/choosing-an-llm-for-arabic/) lists what to test. Side-by-side prices are on the [compare page](https://broadrouter.com/compare/).

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[LLM-as-a-judge](https://broadrouter.com/glossary/llm-as-judge/)

LLM-as-a-judge is an evaluation method in which a language model grades other model outputs, either scoring a single answer against a rubric or picking the better of two answers. It approximates human judgement at much lower cost but has known biases, such as favouring the first or longer answer.

[Eval (LLM evaluation)](https://broadrouter.com/glossary/eval/)

An eval (short for evaluation) is a repeatable test of a language model or prompt: a fixed set of inputs, a description of what a good output looks like, and a grader (code, a human or another model) that scores each output so that versions can be compared.

[Latency](https://broadrouter.com/glossary/latency/)

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

[Prepaid credit](https://broadrouter.com/glossary/prepaid-credit/)

Prepaid credit is money added to an API account before use. Each request deducts its cost from the balance, and when the balance runs out, metered requests stop until you top up. It caps spend at what you have paid in.

[Base URL](https://broadrouter.com/glossary/base-url/)

A base URL is the fixed root address of an API, such as https://api.example.com/v1, that a client puts in front of every endpoint path, such as /chat/completions. In OpenAI-compatible SDKs, changing the base URL, together with the API key and model ID, points the same code at a different provider.

[Fine-tuning](https://broadrouter.com/glossary/fine-tuning/)

Fine-tuning is the process of continuing to train a pre-trained language model on a smaller, task-specific dataset, such as example conversations, so that its weights change and it performs that task or style more reliably without needing long instructions or examples in every prompt.

[OpenAI-compatible API](https://broadrouter.com/glossary/openai-compatible-api/)

An OpenAI-compatible API is an HTTP API that accepts the same request and response formats as the OpenAI API, such as POST /v1/chat/completions, so OpenAI SDKs and tools can use it by changing only the base URL, the API key and the model name.

[Temperature](https://broadrouter.com/glossary/temperature/)

Temperature is a sampling parameter that controls how random a language model's output is. The model divides its token scores by the temperature before turning them into probabilities: low values make the likeliest token dominate, high values flatten the distribution so less likely tokens are chosen more often.

## Questions

### How many examples do I need in an LLM eval set?

Fifty is enough to catch large regressions, 200 to see differences of around 5 to 10 points. Coverage matters more than size, so include every task type and your known failure cases.

### Is LLM-as-a-judge reliable?

Partly. In the MT-Bench study, strong judges agreed with human preferences over 80% of the time, about as often as humans agreed with each other. They also showed position, verbosity and self-enhancement bias, so debias and spot-check them.

### Should the judge model be the same as the model being tested?

No. Judges tend to favour answers that look like their own. Use a judge from a different model family, and keep it pinned to one version for the whole comparison.

### What is position bias in LLM evaluation?

The judge prefers whichever answer appears first, or second, regardless of quality. Fix it by judging each pair twice with the order swapped and counting a win only when both orders agree.

### Should I compare cost and latency in the same eval?

Yes. Log tokens and time for every eval call. A model that scores two points higher at three times the cost or latency may still be the wrong switch.

## Keep reading

-   [How to choose an LLM API provider: a 20-point checklist](https://broadrouter.com/blog/choose-llm-api-provider-checklist/)
-   [Switching to an OpenAI-compatible API: a migration checklist](https://broadrouter.com/blog/openai-compatible-api-migration-checklist/)
-   [Choosing an LLM API for Arabic: what to test](https://broadrouter.com/blog/choosing-an-llm-for-arabic/)
-   [LLM API usage logging: cost per user and feature, PII redacted](https://broadrouter.com/blog/llm-api-usage-logging/)
-   [LLM latency: TTFT vs tokens per second vs end-to-end time](https://broadrouter.com/blog/llm-latency-ttft-throughput/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [BroadNet's GPU hardware and serving approach, as BroadNet describes it](https://broadrouter.com/blog/broadnet-gpu-fleet-and-serving/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)
-   [Guide: Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-evaluate-llm-before-switching&utm_content=page-cta) ·

**Sources**

-   [Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv 2306.05685 (fetched 2026-11-14)](https://arxiv.org/abs/2306.05685)
-   [Arize docs, evaluator best practices (fetched 2026-11-14)](https://arize.com/docs/ax/concepts/evaluators/evaluator-best-practices)
-   [BroadNet Inference API reference (fetched 2026-11-14)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-evaluate-llm-before-switching&utm_content=sources)
-   [BroadNet Inference machine-readable prices, pricing.json (fetched 2026-11-14)](https://inference.broadnet.ai/pricing.json?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-evaluate-llm-before-switching&utm_content=sources)
