BroadRouter
Get started

LLM benchmark

Updated

An LLM benchmark is a fixed, usually public set of tasks with a scoring method, used to measure and compare language models on one skill, such as knowledge questions, coding or speech recognition. A benchmark score shows relative ability on that test, not how a model will do on your own workload.

Also called: model benchmark, benchmark score

Model launches come with tables of benchmark scores. Knowing what a benchmark measures, and what it cannot, keeps you from picking a model on the wrong number.

How it works

A benchmark fixes three things: the tasks, the way the model is prompted, and the scoring rule. Two well-known examples:

Benchmark What it tests Size Scoring
MMLU (2020) Knowledge across “57 tasks including elementary mathematics, US history, computer science, law, and more” 57 subjects Accuracy
SWE-bench (2023) Fixing real GitHub issues by “editing the codebase to address the issue” 2,294 problems from 12 Python repositories Share of issues resolved

Speech has its own: leaderboards rank transcription models by word error rate, where lower is better.

Benchmarks age. When MMLU was published, the largest GPT-3 model improved “over random chance by almost 20 percentage points on average”. When SWE-bench was published, the best model, Claude 2, resolved 1.96% of issues. A benchmark that once separated models well can stop doing so as models improve, so check when a benchmark was made and what scores are typical today.

Why it matters

  • Contamination. The survey by Xu et al. describes benchmark data contamination as models that “inadvertently incorporate evaluation benchmark information from their training data”, leading to “inaccurate or unreliable performance during the evaluation phase”.
  • Narrow coverage. A coding benchmark in Python says little about Arabic customer support. Pick benchmarks that match your task (which leaderboard measures what), then confirm with an eval on your own data.
  • Settings change scores. Prompt format, few-shot examples, reasoning budget and the serving setup all move results. Compare scores only when they were run the same way.
  • Judges add their own bias. Open-ended benchmarks often use an LLM-as-judge; check which judge and rubric were used.

The evaluate an LLM before switching post shows how to build a small test set from your own traffic.

On BroadNet Inference

BroadNet publishes some benchmark numbers of its own; treat them as BroadNet-reported. For BroadNet Transcribe Arabic (based on Cohere Transcribe), BroadNet reports word error rates of 21.1 on MASC dialect speech and 3.8 on FLEURS Arabic, and reports a rank of #2 of 37 (its September 2026 figure; the live board listed it #4 of 39 on 15 November 2026) on the Open Universal Arabic ASR Leaderboard. For Gemma 4 VL 26B, BroadNet reports a blind three-judge evaluation on 240 production conversations in which it matched Claude Sonnet on 5 of 6 steps of BroadNet’s own chatbot pipeline. That is one company’s pipeline, not a general benchmark. Run your own eval before you commit; the $5 starting credit covers Gemma 4 VL 26B and both transcription models.

Questions

What is the difference between a benchmark and an eval?

A benchmark is a shared public test that lets many teams compare models. An eval is your own test set, built from your real prompts and scored on what your product needs. Use benchmarks to shortlist and evals to decide.

What is benchmark contamination?

It happens when test questions, or close copies, end up in a model's training data. The model can then score well by recall rather than skill, so the score overstates its ability.

Are vendor-reported scores reliable?

Treat them as claims. Check which benchmark version, settings and prompts were used, whether anyone reproduced the result, and whether a public leaderboard shows the same thing.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity