BroadRouter
Get started

How to evaluate an LLM before switching providers

By the BroadNet Inference team · 5 min read · Published

Collect 50 to 200 real prompts with known good outcomes. Score both models with code checks first and an LLM judge only where code cannot decide. Swap answer order to cancel position bias, use a judge from a third model family, check it against your own labels, and fix the pass rule, including cost and latency, before you run it.

To evaluate an LLM before switching, run both models on 50 to 200 real prompts with known good outcomes. Score with code checks first and a debiased LLM judge only where code cannot decide, log cost and latency, and fix the pass rule in advance.

The code change itself is small: with an OpenAI-compatible API it is a base URL and a model ID. The migration checklist covers that part, and the provider checklist covers price, limits and privacy. The hard part is knowing whether the new model is as good for your tasks. Public leaderboards do not answer that; Artificial Analysis and its alternatives are for building a shortlist. A small eval set built from your own traffic does, in a day or two of work.

How do you build an eval set from real traffic?

  1. Pull 300 to 1,000 recent requests from your usage logs, after redaction.
  2. Group by task type: extraction, classification, summarisation, chat reply, code edit, tool call.
  3. Sample 10 to 40 per type, more for high-volume or high-risk types. Include long prompts, non-English inputs and the cases users complained about.
  4. Write the expected outcome for each: an exact value, a schema, a test that must pass, or a short rubric (“mentions the refund window; no invented policy”).
  5. Tag critical cases. A wrong invoice total or a leaked instruction is not the same as a slightly worse tone.
  6. Freeze it in version control. Add cases when you find new failures, never delete them to make a model look better.

Which checks should be code and which should be a judge?

Use the cheapest reliable check for each case:

Task Check Judge needed?
JSON extraction Parses, schema validates, fields equal expected No
Classification Label equals expected No
Code edit Tests pass No
Tool call Right function, arguments validate No
Summary Contains required facts; no facts absent from source Sometimes
Chat reply, tone, helpfulness Rubric Yes

Deterministic checks are free, repeatable and impossible to flatter, so push as many cases into code checks as you can.

What goes wrong with LLM-as-a-judge?

An LLM-as-a-judge is a model that grades other models’ answers against a rubric. The MT-Bench study (Zheng et al.) found that strong judges agreed with human preferences more than 80% of the time, about as often as humans agree with each other. The same paper named the biases you have to design around:

Bias What happens Mitigation
Position Prefers the first (or second) answer shown Judge every pair twice, order swapped; count a win only if both agree
Verbosity Prefers longer answers Rubric that rewards correctness and brevity; cap length in the prompt
Self-preference Prefers answers in its own style Judge from a different model family than both candidates
Weak reasoning Misses errors in maths or code Use code checks for those cases instead

Two more rules from practice. Pin the judge: same model version, same prompt, temperature 0, for the whole comparison. And calibrate it: label 30 to 50 pairs yourself, run the judge on them, and only trust it if it agrees with you most of the time. Arize’s evaluator guide gives the same advice (“Collect 20–100 examples” with human labels) and recommends categorical labels such as pass/fail over large numeric scales.

How do you run a side-by-side comparison?

Run both candidates on every case, blind the judge, swap order, and log cost and time:

import os, random, time, json
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
A, B = "broadnet/deepseek-v4.1-flash", "broadnet/qwen3.8-27b"
JUDGE = "broadnet/glm-5.3"  # a third family, pinned

def run(model, case):
    t0 = time.perf_counter()
    r = client.chat.completions.create(model=model, messages=case["messages"], max_tokens=4000, temperature=0)
    return {"text": r.choices[0].message.content or "", "usage": r.usage.model_dump(),
            "secs": time.perf_counter() - t0}

def judge(case, first, second):
    prompt = (f"Task:\n{case['task']}\n\nRubric:\n{case['rubric']}\n\n"
              f"Answer 1:\n{first}\n\nAnswer 2:\n{second}\n\n"
              'Reply with JSON only: {"winner": 1 or 2 or 0 for tie, "reason": "..."}')
    r = client.chat.completions.create(model=JUDGE, temperature=0, max_tokens=2000,
                                       response_format={"type": "json_object"},
                                       messages=[{"role": "user", "content": prompt}])
    return int(json.loads(r.choices[0].message.content or "{}").get("winner", 0))

def compare(case):
    a, b = run(A, case), run(B, case)
    if case.get("check"):                       # deterministic first
        return {"a_ok": case["check"](a["text"]), "b_ok": case["check"](b["text"]), "a": a, "b": b}
    w1 = judge(case, a["text"], b["text"])      # A shown first
    w2 = judge(case, b["text"], a["text"])      # B shown first
    winner = "A" if (w1, w2) == (1, 2) else "B" if (w1, w2) == (2, 1) else "tie"
    return {"winner": winner, "a": a, "b": b}

The swap rule means a model wins only when the judge prefers it in both positions; any disagreement counts as a tie. The share of disagreements is itself useful: a high rate means the judge or the rubric is too vague. Note the max_tokens=4000: on reasoning models, thinking counts toward the limit, and a low cap produces empty answers that look like failures. See why reasoning models return empty replies.

How big a difference is real?

Eval scores are samples. For a pass rate p over n cases, the standard error is √(p × (1 − p) ÷ n), and a 95% margin is about twice that.

Cases Pass rate 95% margin
50 80% about ±11 points
100 80% about ±8 points
200 80% about ±6 points
400 80% about ±4 points

So a model scoring 82% against 79% on 100 cases has not been shown to be better. Look at per-case differences instead: which cases flipped from pass to fail? Those tell you more than the totals, and on critical cases one flip can decide the switch.

What should the decision rule be?

Write it before you run the eval, so the result cannot bend it. A practical template:

  1. No regression on any case tagged critical.
  2. Overall pass rate not lower than the current model’s by more than the margin of error.
  3. p95 latency within your budget; see TTFT vs throughput for how to measure it.
  4. Cost per successful case lower, or higher only by an amount you have agreed is worth the quality. Price the monthly difference with the LLM API cost calculator.
  5. A shadow period: send a share of live traffic to the new model, log both, and compare before you cut over.

The same eval set is the starting point if you are considering a custom model: before fine-tuning covers when prompts and retrieval are enough.

Where BroadNet Inference fits

BroadNet Inference serves several model families behind one OpenAI-compatible endpoint, including DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B, so candidates and a judge from a different family can run through the same client and key. Every response carries token usage for cost comparison, and new accounts start with $5 of credit; without a top-up that credit runs only Gemma 4 VL 26B, so plan a small top-up to compare the larger models. It does not include an evaluation harness or hosted judge tooling; you run the code above yourself. For Arabic workloads, choosing an LLM for Arabic lists what to test. Side-by-side prices are on the compare page.

Questions

How many examples do I need in an LLM eval set?

Fifty is enough to catch large regressions, 200 to see differences of around 5 to 10 points. Coverage matters more than size, so include every task type and your known failure cases.

Is LLM-as-a-judge reliable?

Partly. In the MT-Bench study, strong judges agreed with human preferences over 80% of the time, about as often as humans agreed with each other. They also showed position, verbosity and self-enhancement bias, so debias and spot-check them.

Should the judge model be the same as the model being tested?

No. Judges tend to favour answers that look like their own. Use a judge from a different model family, and keep it pinned to one version for the whole comparison.

What is position bias in LLM evaluation?

The judge prefers whichever answer appears first, or second, regardless of quality. Fix it by judging each pair twice with the order swapped and counting a win only when both orders agree.

Should I compare cost and latency in the same eval?

Yes. Log tokens and time for every eval call. A model that scores two points higher at three times the cost or latency may still be the wrong switch.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity