---
title: "What is LLM-as-a-judge? Model-graded evals explained"
description: "LLM-as-a-judge uses one language model to grade the outputs of another against a rubric. How it works, its known biases, and how to make its scores reliable."
url: https://broadrouter.com/glossary/llm-as-judge/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# LLM-as-a-judge

Updated 2026-11-14

LLM-as-a-judge is an evaluation method in which a language model grades other model outputs, either scoring a single answer against a rubric or picking the better of two answers. It approximates human judgement at much lower cost but has known biases, such as favouring the first or longer answer.

Also called: LLM as a judge, LLM judge, model-graded eval, AI judge, LLM grader

LLM-as-a-judge replaces a human grader with a model call: give it the question, the answer and a rubric, and read back a score.

## How it works

The method was studied in depth by Zheng et al. (2023), who built MT-Bench (multi-turn questions) and Chatbot Arena (crowdsourced head-to-head votes) to test it. Their main result: GPT-4 as a judge agreed with both expert and crowd preferences “at over 80%”, the same level as human-to-human agreement.

There are two common setups:

-   **Single-answer grading.** The judge scores one output on a scale, for example 1 to 5, against written criteria. OpenAI’s graders guide calls this a `score_model` grader that returns “a numeric score based on the prompt within the given range.”
-   **Pairwise comparison.** The judge sees two answers to the same question and picks the better one, or a tie.

A judge prompt usually includes the original question, the answer, a reference answer if you have one, and a rubric. Ask for a short reason before the score, and constrain the score with [structured output](https://broadrouter.com/glossary/structured-output/) so your code can parse it.

## Why it matters

Human grading does not scale to thousands of [eval](https://broadrouter.com/glossary/eval/) cases per prompt change. A judge does, but it brings biases that Zheng et al. measured:

| Bias | What happens | Mitigation |
| --- | --- | --- |
| Position | Prefers the first (or second) answer | Swap order and require agreement |
| Verbosity | Prefers longer answers | Rubric that rewards brevity; length caps |
| Self-enhancement | Prefers its own model’s style | Use a different model as judge |

Judges are also weak at checking maths and facts they cannot verify. For those, give the judge a reference answer, or use code-based checks. A judge that misses [hallucinations](https://broadrouter.com/glossary/hallucination/) gives false confidence.

## On BroadNet Inference

BroadNet does not offer a hosted grading service; you write the judge call yourself against the chat completions endpoint. Judge calls are billed like any request, so a judge reading long answers adds [input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/) per case. Estimate a run with the [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/). If the judge is a reasoning model, leave room in `max_tokens`, because reasoning counts toward it. For the wider process, see [evaluate an LLM before switching](https://broadrouter.com/blog/evaluate-llm-before-switching/).

## Questions

### How accurate is LLM-as-a-judge?

Zheng et al. found GPT-4 as a judge agreed with human preferences over 80% of the time, about the same rate at which humans agree with each other. Check agreement on your own task before trusting it.

### Should the judge be the same model as the one being tested?

Preferably not. Zheng et al. report a self-enhancement bias, and Anthropic's guidance calls a different grading model best practice.

### How do I reduce position bias in pairwise judging?

Run each comparison twice with the answers swapped, and count a win only if the judge picks the same answer both times.

## See also

-   [Eval (LLM evaluation)](https://broadrouter.com/glossary/eval/) (term)
-   [Hallucination](https://broadrouter.com/glossary/hallucination/) (term)
-   [Structured output](https://broadrouter.com/glossary/structured-output/) (term)
-   [Temperature](https://broadrouter.com/glossary/temperature/) (term)
-   [Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/) (term)
-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/) (guide)
-   [How to evaluate an LLM before switching providers](https://broadrouter.com/blog/evaluate-llm-before-switching/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-llm-as-judge&utm_content=page-cta) ·

**Sources**

-   [Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv)](https://arxiv.org/abs/2306.05685)
-   [Anthropic, create strong empirical evaluations (LLM-based grading)](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)
-   [OpenAI, graders guide (score\_model grader)](https://developers.openai.com/api/docs/guides/graders)
