---
title: "Prefill and decode"
type: "glossary term"
url: "https://aifordevs.org/glossary#prefill-and-decode"
markdown: "https://aifordevs.org/glossary/prefill-and-decode.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#prefill-and-decode"
---

# Prefill and decode

Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in parallel and is limited by compute, and decode generates one token per step and is limited by memory bandwidth.

Patel et al. (Splitwise, 2023) call these prompt computation and token generation, and show that token generation underuses compute. Time to first token mostly reflects prefill and time between tokens reflects decode, so the two respond to different fixes.

## Example

Splitwise reports that a 1,500-token prompt on BLOOM-176B takes as long as generating only 6 output tokens.

## Source

[Patel et al., Splitwise: Efficient generative LLM inference using phase splitting (2023)](https://arxiv.org/abs/2311.18677)

## Related terms

- [Inference](https://aifordevs.org/glossary#inference)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [TTFT](https://aifordevs.org/glossary#ttft)

## Read the article

[KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained)

## Cite this term

AI for Devs. "Prefill and decode." https://aifordevs.org/glossary#prefill-and-decode

- Page: https://aifordevs.org/glossary#prefill-and-decode
- Markdown: https://aifordevs.org/glossary/prefill-and-decode.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs