---
title: "KV cache"
type: "glossary term"
url: "https://aifordevs.org/glossary#kv-cache"
markdown: "https://aifordevs.org/glossary/kv-cache.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#kv-cache"
also_known_as: "Key-value cache"
---

# KV cache (Key-value cache)

A KV cache (key-value cache) is memory that stores the attention keys and values of every token already processed, so each new token computes only its own. It trades compute for memory, cutting per-step attention cost from quadratic to linear in sequence length while the cache grows with sequence length and batch size.

Per token, each layer stores one key and one value vector for each KV head, so bytes per token = 2 × layers × KV heads × head dim × bytes per value. The cache belongs to one sequence, so concurrent requests multiply the total. Servers ration that memory with managers such as PagedAttention.

## Example

Llama 3.1 70B has 80 layers, 8 KV heads, and a head dim of 128. At 16-bit values that is 327,680 bytes (320 KiB) per token, or 2.5 GiB for an 8,192-token sequence.

## Source

[Hugging Face Transformers docs: How caching works](https://huggingface.co/docs/transformers/en/cache_explanation)

## Related terms

- [Attention](https://aifordevs.org/glossary#attention)
- [PagedAttention](https://aifordevs.org/glossary#paged-attention)
- [Prefill and decode](https://aifordevs.org/glossary#prefill-and-decode)
- [Inference](https://aifordevs.org/glossary#inference)
- [Grouped-query attention](https://aifordevs.org/glossary#grouped-query-attention)
- [Quantization](https://aifordevs.org/glossary#quantization)
- [Prefix caching](https://aifordevs.org/glossary#prefix-caching)
- [Prompt caching](https://aifordevs.org/glossary#prompt-caching)

## Read the article

[KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained)

## Cite this term

AI for Devs. "KV cache." https://aifordevs.org/glossary#kv-cache

- Page: https://aifordevs.org/glossary#kv-cache
- Markdown: https://aifordevs.org/glossary/kv-cache.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs