---
title: "LLM-as-a-judge"
type: "glossary term"
url: "https://aifordevs.org/glossary#llm-as-a-judge"
markdown: "https://aifordevs.org/glossary/llm-as-a-judge.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#llm-as-a-judge"
---

# LLM-as-a-judge

LLM-as-a-judge is an evaluation method that uses one model to score the outputs of another against a rubric or a comparison prompt. It gives you a scorer for open-ended text that exact-match checks cannot grade.

Zheng et al. (2023) found that strong judges such as GPT-4 agreed with human preferences more than 80% of the time, about the level of agreement among humans. The same paper documents position, verbosity, and self-enhancement biases, so validate a judge against human labels on your own task before you trust it.

## Example

Run the judge twice with the answers swapped. Zheng et al. count a win only when the same answer is preferred in both orders, and call it a tie otherwise.

## Source

[Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)](https://arxiv.org/abs/2306.05685)

## Related terms

- [Eval](https://aifordevs.org/glossary#eval)
- [Golden dataset](https://aifordevs.org/glossary#golden-dataset)

## Read the article

[LLM as a judge: biases, validation, and when to distrust it](https://aifordevs.org/articles/llm-as-a-judge)

## Cite this term

AI for Devs. "LLM-as-a-judge." https://aifordevs.org/glossary#llm-as-a-judge

- Page: https://aifordevs.org/glossary#llm-as-a-judge
- Markdown: https://aifordevs.org/glossary/llm-as-a-judge.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs