LLM-as-a-Judge: How Reliable Are LLM Judges, and How to Evaluate an LLM App or Agent

LLM as a judge means you use one language model to grade the output of another. A strong judge can agree with people about as often as people agree with each other, and it also fails in ways that are now well measured. In the MT-Bench study, GPT-4 agreed with human experts on 85% of non-tie votes, while two experts agreed with each other on 81%. In the same study, GPT-4 kept its verdict on only 65% of answer pairs when the order of the two answers was swapped.

This article collects what peer-reviewed research and the OpenAI, Anthropic and Google evaluation guides say about how reliable LLM judges are, which biases have been measured, and what reduces them. It then covers how to evaluate an LLM app, a RAG pipeline and an agent.

At a glance

Question	Short answer
What is LLM-as-a-judge?	A model grades an output: alone on a scale, against a second output, or against a reference answer.
Can it replace human review?	Partly. Strong judges reach human-level agreement on clear cases. Weaker and smaller judges do not.
Which biases are measured?	Position, length (verbosity), self-preference, leniency, accepting empty answers, and weak grading of math and reasoning.
What reduces them?	Swapping answer order, length control, a reference answer, a clear rubric, and a judge from a different model family.
How do you know your judge works?	Grade a sample by hand and measure agreement with a chance-corrected metric, not only percent agreement.
Is grading an agent different?	Yes. Grade the final state of the environment, run each task several times, and read the full transcript.

What is LLM-as-a-judge?

The term comes from Zheng et al. (NeurIPS 2023 Datasets and Benchmarks). They wanted to rank chat assistants on open-ended questions. Those questions have no single right answer, so string metrics such as BLEU and ROUGE do not work, and human raters are slow and expensive. A strong model can read the question and the answer, and write a score with a reason.

The paper defines three ways to use a judge. Most tools today offer some mix of them.

Mode	What the judge sees	Strength	Weakness reported by Zheng et al.
Pairwise comparison	The question and two answers; it picks one or a tie	Finds small differences between two versions	The number of pairs grows quadratically with the number of systems
Single-answer grading	The question and one answer; it gives a score	Scales to any number of systems	Can miss small differences, and absolute scores move more when the judge model changes
Reference-guided grading	One answer plus a correct reference	Works for questions with a right answer, such as math	Needs a reference for every case

OpenAI's evaluation best practices use the same three types. The guide advises pairwise comparison, classification or pass/fail over open-ended scores, because models are better at choosing between options than at placing an answer on a scale.

How reliable are LLM judges?

Reliability here means agreement with careful human graders. Some studies report percent agreement. Others report a correlation, where 1.0 means the judge ranks outputs exactly as people do and 0 means no relation.

Study	Setting	Result
Zheng et al., NeurIPS 2023	MT-Bench, GPT-4 vs human experts, non-tie votes	85% agreement, against 81% between two humans
Liu et al., G-Eval, EMNLP 2023	Summarization, GPT-4 with chain-of-thought	Spearman correlation of 0.514 with human ratings
Kim et al., Prometheus, ICLR 2024	45 custom score rubrics	Pearson 0.897 for the Prometheus judge, 0.882 for GPT-4, 0.392 for ChatGPT
Thakur et al., GEM workshop at ACL 2025	13 judges grading 9 models on questions with clear answers	Only the best and largest judges align reasonably; scores still differ from human scores by up to 5 points
Zhuge et al., Agent-as-a-Judge, ICML 2025	55 AI development tasks with 365 requirements	An agent judge agreed with human experts about 90% of the time; an LLM judge about 70%

Agreement depends on how different the answers are. Zheng et al. found that GPT-4's agreement with humans rose from 70% to nearly 100% as the gap in quality between two models grew. A judge is most reliable when one answer is clearly better, and least reliable when two answers are close in quality.

Percent agreement can hide large disagreements. Thakur et al. found that judges with high percent agreement could still assign very different scores. They recommend reporting Scott's π next to percent agreement. Scott's π runs up to 100 and subtracts the agreement two graders would reach by chance. On their task, human graders reached a Scott's π of 96.2.

Which biases have been measured in LLM judges?

A bias here means the score changes for a reason that has nothing to do with quality. Ye et al. (ICLR 2025) list 12 such biases and built a framework, CALM, to measure them. The best-documented ones:

Bias	What happens	Measured effect
Position bias	The judge prefers the answer shown first (or second)	GPT-4 gave the same verdict after a swap in 65.0% of cases, GPT-3.5 in 46.2%, Claude-v1 in 23.8% (Zheng et al.)
Position bias, used as an attack	Changing only the order changes the winner	With ChatGPT as judge, Vicuna-13B could be made to beat ChatGPT on 66 of 80 questions (Wang et al., ACL 2024)
Verbosity bias	The judge prefers longer answers	When a list was padded with rephrased copies of its own items, Claude-v1 and GPT-3.5 preferred the padded version 91.3% of the time, GPT-4 8.7% (Zheng et al.)
Self-preference	The judge prefers text it wrote itself	A model's skill at recognizing its own text correlated linearly with how much it favored that text (Panickssery et al., NeurIPS 2024)
Leniency	The judge marks wrong answers as correct	When most judges erred, they erred toward "correct" (Thakur et al.)
Empty answers	Answers such as "Yes" or "Sure" pass	Some judges with high human alignment still accepted them (Thakur et al.)
Weak grading of reasoning	The judge is misled by a wrong worked answer	GPT-4 called a wrong math answer correct in 14 of 20 judgments with the default prompt (Zheng et al.)

Key point: In the MT-Bench study, GPT-4 kept the same verdict after the two answers were swapped in only 65.0% of cases. Run a pairwise judge twice, in both orders, before you trust a win.

Zheng et al. saw GPT-4 favor itself with a 10% higher win rate and Claude-v1 with 25%, but wrote that their data could not confirm the bias. Panickssery et al. later tested self-preference in a controlled experiment and confirmed it.

What reduces LLM judge bias?

Each fix below has a measured effect in the source that proposed it.

Fix	Bias it targets	Evidence
Run a pairwise judge twice with the order swapped; count a win only if both agree	Position	Proposed by Zheng et al.; Wang et al. call it Balanced Position Calibration
Ask the judge to write its evidence before the score	Position, reasoning	Wang et al., Multiple Evidence Calibration; G-Eval uses chain-of-thought for the same reason
Give the judge a reference answer	Weak grading of reasoning	Math failures fell from 14 of 20 to 3 of 20 (Zheng et al.)
Add few-shot judging examples	Position	GPT-4 consistency rose from 65.0% to 77.5%, but the prompts cost 4x more (Zheng et al.)
Control for length in the scoring	Verbosity	AlpacaEval's Spearman correlation with Chatbot Arena rose from 0.94 to 0.98 (Dubois et al., COLM 2024)
Write a specific score rubric	General disagreement	Prometheus reached GPT-4-level correlation with 45 custom rubrics (Kim et al.)
Send unclear cases to a person	All	Wang et al.'s Human-in-the-Loop Calibration sends a person the examples where the judge's verdicts disagree most

Zheng et al. add a warning: higher consistency does not prove higher accuracy, and few-shot examples may add new biases.

How to evaluate an LLM app

LLM evaluation for an application is a different job from ranking public models. You care about your prompts, your users and your failure cases. OpenAI's evaluation guide gives a five-step loop: define the objective, collect a dataset, define metrics, run and compare, and keep evaluating on every change. It names "vibe-based evals" ("it seems like it's working") as an anti-pattern.

Three kinds of grader

Anthropic's Demystifying evals for AI agents (January 2026) sorts graders into three kinds. The same split applies to any LLM app.

Grader	Examples	Strengths	Weaknesses
Code-based	Exact match, unit tests, schema checks, tool-call checks	Fast, cheap, reproducible	Rejects valid answers phrased differently; no nuance
Model-based	Rubric scoring, pairwise comparison, reference-based grading	Handles open-ended output	Not deterministic; needs calibration against people
Human	Expert review, spot checks, A/B tests	Closest to what you want	Slow and expensive

Use a code grader wherever a correct answer can be checked by code. Use a model judge for the rest. Use people to check the model judge. OpenAI's graders guide lists string checks and text similarity before model graders, and says to test a model grader against expert-graded answers.

Offline datasets vs online scoring of live traffic

Most teams need offline and online evaluation, because they answer different questions.

	Offline evaluation	Online evaluation
Question it answers	Is version B better than version A?	Is production getting worse right now?
Input	A fixed dataset of cases	A sample of real requests
Grader	Code, judge, or both, with known answers where possible	Usually a judge, since there is no reference answer
Main risk	The dataset stops matching real traffic	Judge cost, and nobody reads the alerts

Anthropic advises starting with "20-50 simple tasks drawn from real failures", because early changes have large effects that a small set can detect. It also separates capability evals, which should start with a low pass rate, from regression evals, which should pass close to 100% of the time.

How to write a judge rubric

A rubric tells the judge what each score means. The research and the vendor guides agree on a few points.

1. Define every score level. Prometheus was trained and tested with rubrics that describe each score level, and OpenAI's graders guide gives the same advice. Google's Vertex AI evaluation docs offer a static rubric, one fixed set of guidelines for every example, and adaptive rubrics generated from each input prompt.
2. Grade one dimension per judge. Anthropic recommends a separate, isolated judge for each dimension instead of one judge for everything.
3. Let the judge say it cannot tell. Anthropic suggests an "Unknown" option for cases with too little information, so the judge does not invent a grade.
4. Ask for reasons before the score. This is the shared idea behind G-Eval and Multiple Evidence Calibration.
5. Expect the rubric to change. Shankar et al. (UIST 2024) call this "criteria drift": people need criteria to grade outputs, but grading outputs is how they discover their criteria. Write a first rubric, grade some outputs by hand, then revise it. The guide to improving a prompt from feedback shows how to write review comments that a judge can grade against.

RAG evaluation

A retrieval-augmented generation (RAG) app can fail in two places: retrieval finds the wrong passages, or the model ignores the right ones. Ragas (Es et al., EACL 2024 demo track) evaluates both without human-written reference answers. It checks whether the retrieved context is relevant and focused, whether the answer is faithful to that context, and the quality of the answer itself.

The practical point is to score retrieval and generation separately. A single "was the answer good" score cannot tell you which half to fix. For a worked example of grading a RAG agent against user feedback, see the tutorial Improve a RAG agent from user feedback.

AI agent evaluation and agent benchmarks

An agent takes many steps, calls tools and changes the state of the world. That changes what you grade. Anthropic's definitions help:

Term	Meaning
Task	One test with inputs and success criteria
Trial	One attempt at a task; you run several because results vary
Transcript	The full record of a trial: outputs, tool calls, reasoning and intermediate results
Outcome	The final state of the environment, such as whether the booking exists in the database

Three rules follow from the research.

Grade the outcome, not the path. Anthropic warns that checking a fixed order of steps is "overly brittle" and punishes valid solutions. τ-bench (Yao et al., ICLR 2025) grades a customer-service agent by comparing the database at the end of the conversation with an annotated goal state.

Measure consistency, not only success. pass@k is the chance that at least one of k trials succeeds. pass^k ("pass to the power k") is the chance that all k succeed. Anthropic's example: a 75% success rate per trial gives a pass^3 of about 42%. On τ-bench, gpt-4o succeeded on under 50% of tasks, and its pass^8 in the retail domain was under 25%. For a customer-facing agent, pass^k is the number that matters.

Read the transcripts. Anthropic says not to trust scores until someone has read some transcripts. Reading them shows whether a failure is a real agent mistake or a grader that rejected a valid solution. A tracing tool records each step for you; see Using sessions and traces. When the judge itself can read the whole trajectory, agreement improves: Agent-as-a-Judge reached about 90% agreement with human experts, against about 70% for an LLM judge.

How to tell whether your judge can be trusted

Run these checks before you act on a judge's scores, and again after you change the judge model or the rubric.

1. Grade a sample by hand. Take outputs from your own app, not a public benchmark, and have the people who own the product grade them.
2. Measure agreement with a chance-corrected metric. Report Scott's π or a similar metric next to percent agreement, as Thakur et al. recommend.
3. Compare scores, not only verdicts. Thakur et al. found that judges with high percent agreement could still give very different scores.
4. Swap the order. For a pairwise judge, run every pair both ways and count how often the verdict flips.
5. Pad an answer. Add rephrased, useless text to a good answer. If the score goes up, the judge has a length bias.
6. Send empty answers. "Yes", "Sure" and an empty string should fail.
7. Use a judge from a different family from the model under test. Panickssery et al. tie self-preference to self-recognition. A judge from another family should have less of its own style to favor, but none of the cited studies tested a different-family judge directly.

The counterargument: do you need an LLM judge at all?

Simple metrics can rank systems. Thakur et al. found that smaller judges, and even a lexical metric, gave a reasonable signal for ranking the nine models, although they did not match human scores.

Code graders are more stable. Anthropic lists code graders as fast, cheap and reproducible. A judge call costs money, varies between runs and needs its own checking.

Judges are weakest on small differences. Zheng et al. found agreement near 70% when two answers were close in quality and near 100% when one was clearly better. A small change to a prompt usually produces two answers that are close.

Each situation has a different default:

Situation	Better default
The answer can be checked by code	A code grader
The answer is open-ended and quality is subjective	A judge with a rubric, checked against human grades
Two versions are close	Pairwise grading in both orders, plus human review of the disagreements
Production traffic with no reference answer	A sampled online judge, with alerts and regular human spot checks
An agent that changes state	A check on the final state, over several trials

Start at the top and stop at the first question you answer yes to.

How AcruxCore handles LLM evaluation

AcruxCore's evaluation is built on single-answer grading, and it also stores code scores and human feedback.

• You choose the judge model. There is no default. Every run and rule names a model registered under Gateway → Models. The judge call runs at temperature 0, under your normal budgets and rate limits, and gets its own trace.
• The built-in judge returns a score from 0 to 100, a pass/fail flag and a reason. It reads the output, the example's criteria, the dataset's overall feedback, and the earlier turns of a session. The output is marked as data, not instructions. An unparseable reply is retried once, then stored unscored with a reason.
• You can replace the rubric. A custom judge prompt is a versioned prompt, and a run freezes the version it started with.
• Offline runs. An experiment runs every prompt version against every model over a dataset, and a compare view shows which examples improved or regressed. Local runs run your own app and code scorers through the SDK, against a reference answer stored in each example.
• Online rules. An evaluation rule judges a sample of live LLM calls, 10% and at most 500 a day by default. It can notify the team's owners when a score falls below a threshold, and turn low-scoring calls into a dataset.
• Feedback. Thumbs, labels and comments on traces can build a dataset in which each comment becomes the grading criteria.

Problem from the research	What AcruxCore does
Judge quality depends on the model	You pick the judge model per run and per rule
Vague criteria	Per-example criteria, overall feedback, or your own versioned rubric
Judges miss what people know	Feedback comments become grading criteria
Code can check some answers better	Code scorers with named scores in local runs
Production drifts	Sampled online rules with a score alert

AcruxCore does not have four things that this article discusses. There is no pairwise judge, so there is no position-swap check. AcruxCore does not compute agreement between judge scores and human feedback for you. The judge is not sent the example's input or its expected answer, so a reference answer for the judge has to be written into the criteria or the rubric. Online rules grade one LLM call at a time, including its tool calls, and do not grade a whole agent trajectory.

For how other evaluation platforms handle judges, datasets and self-hosting, see Braintrust alternatives for LLM evals.

A checklist for LLM evaluation

• Start with 20 to 50 cases taken from real failures
• Use a code grader wherever code can decide the answer
• Define every score level in the rubric, one dimension per judge
• Grade a sample by hand and measure agreement with a chance-corrected metric
• Run pairwise judges in both orders, and test with padded and empty answers
• Score retrieval and generation separately in a RAG app
• For agents, grade the final state and report pass^k over repeated trials
• Sample live traffic with an online judge, and read the low scores every week

The open question

Most judge studies measure agreement on public benchmarks with clear answers, often with GPT-4 as the judge. Your app has its own criteria, which Shankar et al. show you only discover by grading. Your app also has cases where two answers are close in quality, and Zheng et al. found agreement is lowest there. The open question is how small a human-graded sample can be and still tell you that a judge agrees with your team on your traffic. Until research answers that for product data, the safe default is to keep a person reading a regular sample of the judge's scores.

Sources

Research papers (peer-reviewed)

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS Datasets and Benchmarks Track, 2023
• Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, EMNLP, 2023
• Kim et al., Prometheus: Inducing Fine-grained Evaluation Capability in Language Models, ICLR, 2024
• Wang et al., Large Language Models are not Fair Evaluators, ACL, 2024
• Panickssery, Bowman and Feng, LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS, 2024
• Dubois et al., Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators, COLM, 2024
• Es et al., RAGAs: Automated Evaluation of Retrieval Augmented Generation, EACL System Demonstrations, 2024
• Shankar et al., Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences, UIST, 2024
• Ye et al., Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge, ICLR, 2025
• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, ICLR, 2025
• Zhuge et al., Agent-as-a-Judge: Evaluate Agents with Agents, ICML, 2025
• Thakur et al., Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges, Fourth Workshop on Generation, Evaluation and Metrics (GEM²) at ACL, 2025

Engineering guides

• Anthropic, Demystifying evals for AI agents, January 2026
• OpenAI, Evaluation best practices
• OpenAI, Graders
• Google Cloud, Define your evaluation metrics, Vertex AI documentation