Topic

Evals and observability

Measuring whether a model or agent is good enough for your task, and seeing what it did when it was not.

Benchmarks, tracing, LLM-as-judge02 articles06 glossary terms03 platforms

Articles

Read on this topic

Deep dives and guides filed under this topic, newest first.

All articles

Deep dive

LLM as a judge: biases, validation, and when to distrust it

LLM as a judge means grading model output with a model. See what Zheng et al. measured, where bias enters, and how to validate a judge against human labels.

Nov 08 202612 min readEvals and observability

Guide

LLM evaluation: build an eval suite and gate deploys in CI

LLM evaluation, step by step: define failure modes, build a golden dataset, choose metrics, run an offline suite, compare to a baseline, and gate CI.

Nov 08 202611 min readEvals and observability

Glossary

Terms you will meet

Eval: An eval is a repeatable test of a model or an agent on examples from your task, scored by code, by people, or by another model. Evals show whether a prompt change, a model swap, or a new tool made results better or worse.
Golden dataset: A golden dataset is a fixed, versioned set of inputs with known-good outputs or success criteria that you use to score a model or an agent. Scores stay comparable across runs because the set does not change between them.
Guardrails: Guardrails are checks that run on a model's input or output to block, rewrite, or flag content such as personal data, prompt injection, policy violations, or malformed responses. They run outside the model, so they still apply when the model's output is wrong.
Hallucination: A hallucination is model output that reads as confident and fluent but is false or unsupported by the input, such as an invented API, citation, or number. Retrieval and tool use reduce it, and evals measure it, but none of them remove it.
LLM-as-a-judge: LLM-as-a-judge is an evaluation method that uses one model to score the outputs of another against a rubric or a comparison prompt. It gives you a scorer for open-ended text that exact-match checks cannot grade.
SWE-bench: SWE-bench is a benchmark that asks a model or agent to fix real GitHub issues by editing the repository, then runs the project's tests to check the patch. The original set has 2,294 problems from 12 Python repositories.

Platforms

Tools in this part of the stack

Full directory