To evaluate an LLM before switching, run both models on 50 to 200 real prompts with known good outcomes. Score with code checks first and a debiased LLM judge only where code cannot decide, log cost and latency, and fix the pass rule in advance.
The code change itself is small: with an OpenAI-compatible API it is a base URL and a model ID. The migration checklist covers that part, and the provider checklist covers price, limits and privacy. The hard part is knowing whether the new model is as good for your tasks. Public leaderboards do not answer that; Artificial Analysis and its alternatives are for building a shortlist. A small eval set built from your own traffic does, in a day or two of work.
How do you build an eval set from real traffic?
- Pull 300 to 1,000 recent requests from your usage logs, after redaction.
- Group by task type: extraction, classification, summarisation, chat reply, code edit, tool call.
- Sample 10 to 40 per type, more for high-volume or high-risk types. Include long prompts, non-English inputs and the cases users complained about.
- Write the expected outcome for each: an exact value, a schema, a test that must pass, or a short rubric (“mentions the refund window; no invented policy”).
- Tag critical cases. A wrong invoice total or a leaked instruction is not the same as a slightly worse tone.
- Freeze it in version control. Add cases when you find new failures, never delete them to make a model look better.
Which checks should be code and which should be a judge?
Use the cheapest reliable check for each case:
| Task | Check | Judge needed? |
|---|---|---|
| JSON extraction | Parses, schema validates, fields equal expected | No |
| Classification | Label equals expected | No |
| Code edit | Tests pass | No |
| Tool call | Right function, arguments validate | No |
| Summary | Contains required facts; no facts absent from source | Sometimes |
| Chat reply, tone, helpfulness | Rubric | Yes |
Deterministic checks are free, repeatable and impossible to flatter, so push as many cases into code checks as you can.
What goes wrong with LLM-as-a-judge?
An LLM-as-a-judge is a model that grades other models’ answers against a rubric. The MT-Bench study (Zheng et al.) found that strong judges agreed with human preferences more than 80% of the time, about as often as humans agree with each other. The same paper named the biases you have to design around:
| Bias | What happens | Mitigation |
|---|---|---|
| Position | Prefers the first (or second) answer shown | Judge every pair twice, order swapped; count a win only if both agree |
| Verbosity | Prefers longer answers | Rubric that rewards correctness and brevity; cap length in the prompt |
| Self-preference | Prefers answers in its own style | Judge from a different model family than both candidates |
| Weak reasoning | Misses errors in maths or code | Use code checks for those cases instead |
Two more rules from practice. Pin the judge: same model version, same prompt, temperature 0, for the whole comparison. And calibrate it: label 30 to 50 pairs yourself, run the judge on them, and only trust it if it agrees with you most of the time. Arize’s evaluator guide gives the same advice (“Collect 20–100 examples” with human labels) and recommends categorical labels such as pass/fail over large numeric scales.
How do you run a side-by-side comparison?
Run both candidates on every case, blind the judge, swap order, and log cost and time:
import os, random, time, json
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
A, B = "broadnet/deepseek-v4.1-flash", "broadnet/qwen3.8-27b"
JUDGE = "broadnet/glm-5.3" # a third family, pinned
def run(model, case):
t0 = time.perf_counter()
r = client.chat.completions.create(model=model, messages=case["messages"], max_tokens=4000, temperature=0)
return {"text": r.choices[0].message.content or "", "usage": r.usage.model_dump(),
"secs": time.perf_counter() - t0}
def judge(case, first, second):
prompt = (f"Task:\n{case['task']}\n\nRubric:\n{case['rubric']}\n\n"
f"Answer 1:\n{first}\n\nAnswer 2:\n{second}\n\n"
'Reply with JSON only: {"winner": 1 or 2 or 0 for tie, "reason": "..."}')
r = client.chat.completions.create(model=JUDGE, temperature=0, max_tokens=2000,
response_format={"type": "json_object"},
messages=[{"role": "user", "content": prompt}])
return int(json.loads(r.choices[0].message.content or "{}").get("winner", 0))
def compare(case):
a, b = run(A, case), run(B, case)
if case.get("check"): # deterministic first
return {"a_ok": case["check"](a["text"]), "b_ok": case["check"](b["text"]), "a": a, "b": b}
w1 = judge(case, a["text"], b["text"]) # A shown first
w2 = judge(case, b["text"], a["text"]) # B shown first
winner = "A" if (w1, w2) == (1, 2) else "B" if (w1, w2) == (2, 1) else "tie"
return {"winner": winner, "a": a, "b": b}
The swap rule means a model wins only when the judge prefers it in both positions; any disagreement counts as a tie. The share of disagreements is itself useful: a high rate means the judge or the rubric is too vague. Note the max_tokens=4000: on reasoning models, thinking counts toward the limit, and a low cap produces empty answers that look like failures. See why reasoning models return empty replies.
How big a difference is real?
Eval scores are samples. For a pass rate p over n cases, the standard error is √(p × (1 − p) ÷ n), and a 95% margin is about twice that.
| Cases | Pass rate | 95% margin |
|---|---|---|
| 50 | 80% | about ±11 points |
| 100 | 80% | about ±8 points |
| 200 | 80% | about ±6 points |
| 400 | 80% | about ±4 points |
So a model scoring 82% against 79% on 100 cases has not been shown to be better. Look at per-case differences instead: which cases flipped from pass to fail? Those tell you more than the totals, and on critical cases one flip can decide the switch.
What should the decision rule be?
Write it before you run the eval, so the result cannot bend it. A practical template:
- No regression on any case tagged critical.
- Overall pass rate not lower than the current model’s by more than the margin of error.
- p95 latency within your budget; see TTFT vs throughput for how to measure it.
- Cost per successful case lower, or higher only by an amount you have agreed is worth the quality. Price the monthly difference with the LLM API cost calculator.
- A shadow period: send a share of live traffic to the new model, log both, and compare before you cut over.
The same eval set is the starting point if you are considering a custom model: before fine-tuning covers when prompts and retrieval are enough.
Where BroadNet Inference fits
BroadNet Inference serves several model families behind one OpenAI-compatible endpoint, including DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B, so candidates and a judge from a different family can run through the same client and key. Every response carries token usage for cost comparison, and new accounts start with $5 of credit; without a top-up that credit runs only Gemma 4 VL 26B, so plan a small top-up to compare the larger models. It does not include an evaluation harness or hosted judge tooling; you run the code above yourself. For Arabic workloads, choosing an LLM for Arabic lists what to test. Side-by-side prices are on the compare page.
