---
title: "What is an LLM benchmark? Scores, limits and contamination"
description: "An LLM benchmark is a fixed public test set used to score and compare models, such as MMLU or SWE-bench. How benchmarks work, and why a score is not your eval."
url: https://broadrouter.com/glossary/llm-benchmark/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# LLM benchmark

Updated 2026-11-16

An LLM benchmark is a fixed, usually public set of tasks with a scoring method, used to measure and compare language models on one skill, such as knowledge questions, coding or speech recognition. A benchmark score shows relative ability on that test, not how a model will do on your own workload.

Also called: model benchmark, benchmark score

Model launches come with tables of benchmark scores. Knowing what a benchmark measures, and what it cannot, keeps you from picking a model on the wrong number.

## How it works

A benchmark fixes three things: the tasks, the way the model is prompted, and the scoring rule. Two well-known examples:

| Benchmark | What it tests | Size | Scoring |
| --- | --- | --- | --- |
| MMLU (2020) | Knowledge across “57 tasks including elementary mathematics, US history, computer science, law, and more” | 57 subjects | Accuracy |
| SWE-bench (2023) | Fixing real GitHub issues by “editing the codebase to address the issue” | 2,294 problems from 12 Python repositories | Share of issues resolved |

Speech has its own: leaderboards rank transcription models by [word error rate](https://broadrouter.com/glossary/word-error-rate/), where lower is better.

Benchmarks age. When MMLU was published, the largest GPT-3 model improved “over random chance by almost 20 percentage points on average”. When SWE-bench was published, the best model, Claude 2, resolved 1.96% of issues. A benchmark that once separated models well can stop doing so as models improve, so check when a benchmark was made and what scores are typical today.

## Why it matters

-   **Contamination.** The survey by Xu et al. describes benchmark data contamination as models that “inadvertently incorporate evaluation benchmark information from their training data”, leading to “inaccurate or unreliable performance during the evaluation phase”.
-   **Narrow coverage.** A coding benchmark in Python says little about Arabic customer support. Pick benchmarks that match your task ([which leaderboard measures what](https://broadrouter.com/blog/artificial-analysis-alternatives/)), then confirm with an [eval](https://broadrouter.com/glossary/eval/) on your own data.
-   **Settings change scores.** Prompt format, few-shot examples, reasoning budget and the serving setup all move results. Compare scores only when they were run the same way.
-   **Judges add their own bias.** Open-ended benchmarks often use an [LLM-as-judge](https://broadrouter.com/glossary/llm-as-judge/); check which judge and rubric were used.

The [evaluate an LLM before switching](https://broadrouter.com/blog/evaluate-llm-before-switching/) post shows how to build a small test set from your own traffic.

## On BroadNet Inference

BroadNet publishes some benchmark numbers of its own; treat them as BroadNet-reported. For [BroadNet Transcribe Arabic](https://broadrouter.com/models/cohere-transcribe-arabic/) (based on Cohere Transcribe), BroadNet reports word error rates of 21.1 on MASC dialect speech and 3.8 on FLEURS Arabic, and reports a rank of #2 of 37 (its September 2026 figure; the live board listed it #4 of 39 on 15 November 2026) on the Open Universal Arabic ASR Leaderboard. For [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/), BroadNet reports a blind three-judge evaluation on 240 production conversations in which it matched Claude Sonnet on 5 of 6 steps of BroadNet’s own chatbot pipeline. That is one company’s pipeline, not a general benchmark. Run your own eval before you commit; the $5 starting credit covers Gemma 4 VL 26B and both transcription models.

## Questions

### What is the difference between a benchmark and an eval?

A benchmark is a shared public test that lets many teams compare models. An eval is your own test set, built from your real prompts and scored on what your product needs. Use benchmarks to shortlist and evals to decide.

### What is benchmark contamination?

It happens when test questions, or close copies, end up in a model's training data. The model can then score well by recall rather than skill, so the score overstates its ability.

### Are vendor-reported scores reliable?

Treat them as claims. Check which benchmark version, settings and prompts were used, whether anyone reproduced the result, and whether a public leaderboard shows the same thing.

## See also

-   [Eval (LLM evaluation)](https://broadrouter.com/glossary/eval/) (term)
-   [LLM-as-a-judge](https://broadrouter.com/glossary/llm-as-judge/) (term)
-   [Word error rate](https://broadrouter.com/glossary/word-error-rate/) (term)
-   [Hallucination](https://broadrouter.com/glossary/hallucination/) (term)
-   [Open-weight model](https://broadrouter.com/glossary/open-weight-model/) (term)
-   [How to evaluate an LLM before switching providers](https://broadrouter.com/blog/evaluate-llm-before-switching/) (article)
-   [Artificial Analysis alternatives: where to compare LLMs in 2026](https://broadrouter.com/blog/artificial-analysis-alternatives/) (article)
-   [Choosing an LLM API for Arabic: what to test](https://broadrouter.com/blog/choosing-an-llm-for-arabic/) (article)
-   [Arabic speech-to-text API: how to choose one in 2026](https://broadrouter.com/blog/arabic-speech-to-text-api-buyers-guide/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-llm-benchmark&utm_content=page-cta) ·

**Sources**

-   [Hendrycks et al., Measuring Massive Multitask Language Understanding (MMLU), arXiv 2009.03300 (checked 2026-11-15)](https://arxiv.org/abs/2009.03300)
-   [Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, arXiv 2310.06770 (checked 2026-11-15)](https://arxiv.org/abs/2310.06770)
-   [Xu et al., Benchmark Data Contamination of Large Language Models: A Survey, arXiv 2406.04244 (checked 2026-11-15)](https://arxiv.org/abs/2406.04244)
-   [BroadNet, BroadNet Transcribe Arabic model page: reported WER and leaderboard rank (checked 2026-11-16)](https://www.broadnet.ai/inference/models/cohere-transcribe-arabic)
