---
title: "Evals and observability"
type: "topic"
url: "https://aifordevs.org/topics/evals"
markdown: "https://aifordevs.org/topics/evals.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/topics/evals"
---

# Evals and observability

Measuring whether a model or agent is good enough for your task, and seeing what it did when it was not.

## Articles

- [LLM as a judge: biases, validation, and when to distrust it](https://aifordevs.org/articles/llm-as-a-judge.md): LLM as a judge means grading model output with a model. See what Zheng et al. measured, where bias enters, and how to validate a judge against human labels. (2026-11-08)
- [LLM evaluation: build an eval suite and gate deploys in CI](https://aifordevs.org/articles/llm-evaluation.md): LLM evaluation, step by step: define failure modes, build a golden dataset, choose metrics, run an offline suite, compare to a baseline, and gate CI. (2026-11-08)

## Glossary terms

- [Eval](https://aifordevs.org/glossary/eval.md): An eval is a repeatable test of a model or an agent on examples from your task, scored by code, by people, or by another model. Evals show whether a prompt change, a model swap, or a new tool made results better or worse.
- [Golden dataset](https://aifordevs.org/glossary/golden-dataset.md): A golden dataset is a fixed, versioned set of inputs with known-good outputs or success criteria that you use to score a model or an agent. Scores stay comparable across runs because the set does not change between them.
- [Guardrails](https://aifordevs.org/glossary/guardrails.md): Guardrails are checks that run on a model's input or output to block, rewrite, or flag content such as personal data, prompt injection, policy violations, or malformed responses. They run outside the model, so they still apply when the model's output is wrong.
- [Hallucination](https://aifordevs.org/glossary/hallucination.md): A hallucination is model output that reads as confident and fluent but is false or unsupported by the input, such as an invented API, citation, or number. Retrieval and tool use reduce it, and evals measure it, but none of them remove it.
- [LLM-as-a-judge](https://aifordevs.org/glossary/llm-as-a-judge.md): LLM-as-a-judge is an evaluation method that uses one model to score the outputs of another against a rubric or a comparison prompt. It gives you a scorer for open-ended text that exact-match checks cannot grade.
- [SWE-bench](https://aifordevs.org/glossary/swe-bench.md): SWE-bench is a benchmark that asks a model or agent to fix real GitHub issues by editing the repository, then runs the project's tests to check the patch. The original set has 2,294 problems from 12 Python repositories.

## Platforms

- [Langfuse](https://aifordevs.org/platforms/langfuse.md): Open-source LLM engineering platform for tracing, prompt management, and evaluations.
- [Maxim AI](https://aifordevs.org/platforms/maxim-ai.md): Platform for simulating, evaluating, and observing AI agents before and after release.
- [Phoenix](https://aifordevs.org/platforms/arize-phoenix.md): Source-available tracing and evaluation tool built on OpenTelemetry, which you can run locally.