---
title: "Golden dataset"
type: "glossary term"
url: "https://aifordevs.org/glossary#golden-dataset"
markdown: "https://aifordevs.org/glossary/golden-dataset.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#golden-dataset"
---

# Golden dataset

A golden dataset is a fixed, versioned set of inputs with known-good outputs or success criteria that you use to score a model or an agent. Scores stay comparable across runs because the set does not change between them.

Build it from real usage and real failures, then add a case each time a bug reaches users. Anthropic's eval guidance says 20 to 50 simple tasks drawn from real failures is a great start, and that teams often wait too long for hundreds. Keep it separate from the examples you tune prompts against, or the score stops predicting production.

## Example

A support agent's set might hold 40 past tickets, each with the correct policy, the expected tool call, and a grader that checks both.

## Source

[Anthropic, Demystifying evals for AI agents (2026)](https://www.anthropic.com/research/demystifying-evals-for-ai-agents)

## Related terms

- [Eval](https://aifordevs.org/glossary#eval)
- [LLM-as-a-judge](https://aifordevs.org/glossary#llm-as-a-judge)
- [SWE-bench](https://aifordevs.org/glossary#swe-bench)

## Read the article

[LLM evaluation: build an eval suite and gate deploys in CI](https://aifordevs.org/articles/llm-evaluation)

## Cite this term

AI for Devs. "Golden dataset." https://aifordevs.org/glossary#golden-dataset

- Page: https://aifordevs.org/glossary#golden-dataset
- Markdown: https://aifordevs.org/glossary/golden-dataset.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs