---
title: "Speculative decoding"
type: "glossary term"
url: "https://aifordevs.org/glossary#speculative-decoding"
markdown: "https://aifordevs.org/glossary/speculative-decoding.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#speculative-decoding"
---

# Speculative decoding

Speculative decoding is a decoding method where a fast draft step proposes several tokens and the large model verifies them in one forward pass, keeping the longest accepted prefix. With the standard acceptance rule, the output distribution matches the large model's.

Leviathan et al. (2023) accept each draft token with probability min(1, p / q), where p is the target model's probability and q is the draft's. A rejection discards the later drafts and resamples from an adjusted distribution. The target model never needs more serial steps than standard decoding, and the speedup grows with the acceptance rate.

## Example

With 4 drafts per step and a 70% acceptance rate, the expected yield is (1 - 0.7^5) / (1 - 0.7), or about 2.8 tokens per target-model call. Leviathan et al. report 2x to 3x faster decoding on T5-XXL.

## Source

[Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2023)](https://arxiv.org/abs/2211.17192)

## Related terms

- [Inference](https://aifordevs.org/glossary#inference)
- [Throughput](https://aifordevs.org/glossary#throughput)

## Read the article

[Speculative decoding: how drafts speed up LLM inference](https://aifordevs.org/articles/speculative-decoding)

## Cite this term

AI for Devs. "Speculative decoding." https://aifordevs.org/glossary#speculative-decoding

- Page: https://aifordevs.org/glossary#speculative-decoding
- Markdown: https://aifordevs.org/glossary/speculative-decoding.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs