---
title: "Constrained decoding"
type: "glossary term"
url: "https://aifordevs.org/glossary#constrained-decoding"
markdown: "https://aifordevs.org/glossary/constrained-decoding.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#constrained-decoding"
---

# Constrained decoding

Constrained decoding is generation in which tokens that would break a grammar or schema are masked at each step before sampling. The output follows the grammar by construction, unless generation stops early.

Willard and Louf (2023) frame it as walking a finite-state machine: precompute which vocabulary tokens are legal in each state, then mask the rest. The schema has to be compiled first, so the first request with a new schema is slower. Anthropic's docs describe compiling JSON Schemas into grammars and caching them for 24 hours after last use.

## Example

With a field limited to "yes" or "no", every token that cannot extend one of those two strings gets probability zero at that position.

## Source

[Willard and Louf, Efficient Guided Generation for Large Language Models (2023)](https://arxiv.org/abs/2307.09702)

## Related terms

- [Structured outputs](https://aifordevs.org/glossary#structured-outputs)
- [Tool calling](https://aifordevs.org/glossary#tool-calling)

## Cite this term

AI for Devs. "Constrained decoding." https://aifordevs.org/glossary#constrained-decoding

- Page: https://aifordevs.org/glossary#constrained-decoding
- Markdown: https://aifordevs.org/glossary/constrained-decoding.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs