KV cache explained: what it stores and what it costs

A KV cache is the block of GPU memory that holds the attention keys and values a transformer computed for every earlier token in a sequence. An LLM reads it on every generated token, because each new token attends to all of those earlier positions. Without it, token 1,001 would repeat the key and value work for the previous 1,000 tokens in every layer. The cache removes that repeated math and charges you in memory, and that memory is what caps how many users fit on a GPU.

#What is a KV cache

A KV cache stores a key tensor and a value tensor per KV head, per layer, for every token, so later tokens read them instead of recomputing them. Keys and values for earlier positions do not change when a new token arrives, so caching them changes speed and memory without changing the math.

Attention projects each token into a query, a key, and a value, the scheme Vaswani et al. introduced in 2017. To produce the next token, each layer compares the newest query against the keys of all earlier tokens and mixes their values.

Only the newest token needs a new query. Everything else a transformer layer needs from earlier positions is already sitting in the cache.

Scope is where people mix up the terms. The cache lives inside one generation. Prompt caching keeps part of it alive across separate requests, and the section below ties the two together.

#How the KV cache works in prefill and decode

The cache is filled during prefill and read during decode. Prefill computes keys and values for the whole prompt in one parallel pass and writes them to the cache. Decode then generates one token per step, reading every cached entry and appending one new entry per layer until the response ends.

Prefill and decode load the hardware differently. Prefill is a large batch of matrix multiplications, and the first output token cannot appear until it finishes, which makes prefill time a floor under time to first token.

Decode does little arithmetic per token but moves a lot of data. Shazeer named this problem in 2019: repeatedly loading large key and value tensors eats memory bandwidth. The cost grows with every token in the sequence, so decode slows as a conversation gets longer.

#How big a KV cache gets

KV cache size is one multiplication from a model's config: two tensors, times layers, times KV heads, times head dimension, times bytes per value. Llama 3.1 70B comes to 320 KiB per token at 16 bits, which becomes 2.5 GiB for an 8K-token sequence and 40 GiB at 128K tokens, per sequence.

bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_value

The 2 counts keys and values, and at 16 bits bytes_per_value is 2. head_dim is the model dimension divided by the attention heads. Architecture numbers below come from Table 3 of the Llama 3 paper, recomputed with code on November 8, 2026.

Here 8K means 8,192 tokens and 128K means 131,072 tokens. Sixteen Llama 3.1 70B sequences at 8K tokens need 40 GiB of cache on top of 141 GB of 16-bit weights.

The same arithmetic runs against any Hugging Face config.json. This script ran on Python 3.15.1 and handles the compressed cache format DeepSeek models use.

"""Size a KV cache from a Hugging Face config.json. Tested with Python 3.15."""
import json
import sys
 
 
def kv_bytes_per_token(config_path: str, bytes_per_value: int = 2) -> int:
    cfg = json.load(open(config_path))
    layers = cfg["num_hidden_layers"]
    # DeepSeek-style MLA caches one compressed latent, not per-head K and V.
    if cfg.get("kv_lora_rank"):
        latent = cfg["kv_lora_rank"] + cfg["qk_rope_head_dim"]
        return layers * latent * bytes_per_value
    kv_heads = cfg.get("num_key_value_heads") or cfg["num_attention_heads"]
    head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]
    return 2 * layers * kv_heads * head_dim * bytes_per_value
 
 
if __name__ == "__main__":
    for path in sys.argv[1:]:
        per_token = kv_bytes_per_token(path)
        name = path.rsplit("/", 1)[-1]
        print(f"{name}: {per_token / 1024:.1f} KiB per token, "
              f"{per_token * 8192 / 2**30:.2f} GiB per 8K-token sequence")

Meta's own Llama 3.1 repository is gated, so the run below reads NousResearch's public copy, whose layer and head counts match Table 3. Qwen3-8B and DeepSeek-V3 publish theirs openly.

curl -sL -o qwen3-8b.json https://huggingface.co/Qwen/Qwen3-8B/resolve/main/config.json
curl -sL -o llama-3.1-8b.json https://huggingface.co/NousResearch/Meta-Llama-3.1-8B/resolve/main/config.json
curl -sL -o deepseek-v3.json https://huggingface.co/deepseek-ai/DeepSeek-V3/resolve/main/config.json
python3 kv_size.py qwen3-8b.json llama-3.1-8b.json deepseek-v3.json

qwen3-8b.json: 144.0 KiB per token, 1.12 GiB per 8K-token sequence
llama-3.1-8b.json: 128.0 KiB per token, 1.00 GiB per 8K-token sequence
deepseek-v3.json: 68.6 KiB per token, 0.54 GiB per 8K-token sequence

Qwen3-8B has 36 layers to Llama 3.1 8B's 32, which accounts for its extra 16 KiB per token.

#When the standard KV cache formula is wrong

The standard formula overstates any model that does not store a key and value per head in every layer. DeepSeek-V3 caches one compressed latent per layer, and gpt-oss-120b alternates sliding-window and full-attention layers, so a sizing script has to read those config fields before it trusts the textbook math.

DeepSeek introduced multi-head latent attention (MLA) in its V2 report. MLA caches one compressed vector plus a small RoPE key, so per-token cache grows with kv_lora_rank + qk_rope_head_dim per layer instead of 2 x kv_heads x head_dim.

Feed DeepSeek-V3's config through the naive formula and you get 1.67 MiB per token. The MLA branch in the script returns 68.6 KiB, about 25 times less. vLLM's cache spec treats MLA the same way, with a single latent per state and no separate value tensor.

Sliding-window layers cap the cache at the window size. The gpt-oss-120b config alternates 18 sliding-window layers with 18 full-attention layers and a 128-token window.

Counting all 36 layers as full attention gives 72 KiB per token. Counting only the 18 full layers gives 36 KiB, so the formula overstates a long sequence by about two times.

Vendors keep adding variants. MiniMax says its M3 sparse attention reduces attention compute and memory compared with GQA, which is the vendor's own claim. Read the model card before you trust any single formula.

#What concurrency costs on a real GPU

Concurrency cost equals cache per sequence times active sequences, paid from whatever memory the weights leave. A 141 GB H200 running Llama 3.1 8B at 16 bits leaves about 106 GiB for the cache. That holds about 105 sequences at 8K tokens and about 6 at 128K tokens.

Counts are full-length sequences held at once, recomputed with code on November 8, 2026. The KV pool is GPU memory times vLLM's default gpu_memory_utilization of 0.92 for a fixed-size cache, minus weights. GPU sizes come from NVIDIA's H100 and H200 pages, and weight sizes from each model's Hugging Face parameter count.

Going from 8K to 128K tokens cuts concurrency by 16 times, because the cache grows linearly with sequence length. Real traffic runs shorter than the maximum, which raises these counts, and activation memory lowers them.

Quantization moves the budget more than any other row.

FP8 weights and an FP8 cache fit Llama 3.1 70B on one H200 with the same 44-sequence capacity as two H200s at 16 bits. That only helps if quantization holds quality on your task. A mixture-of-experts model goes the other way, with far more memory spent on loaded weights and less left for the cache.

#Why decode is memory-bound

Decode is memory-bound because each step reads the weights and every active sequence's cache from GPU memory while doing little arithmetic per token. Batching shares the weight read across sequences but not the cache read, so cache traffic can pass weight traffic as contexts grow.

A rough floor shows the split. An H100 moves 3.35 TB/s of GPU memory, so reading Llama 3.1 8B's 16.1 GB of 16-bit weights once takes 4.8 ms. One 8K-token sequence's 1 GiB cache adds 0.32 ms, and a batch of 32 such sequences adds 10.3 ms, twice the weight read.

At 128K tokens, the same batch would spend 164 ms per step on cache reads. Those figures are lower bounds from capacity and bandwidth alone, and they ignore DRAM efficiency and on-package caches. No benchmark sits behind them.

Published measurements point the same way. Recasens et al. report that large-batch GPU inference stays limited by DRAM bandwidth saturation, leaving compute capacity unused. DeepSeek's V3 report says decoding in its own fleet is bottlenecked by memory access rather than computation.

Speculative decoding exists to spend that idle compute. A long-context request batched with short ones slows all of them, because each step waits on every cache read.

#How serving engines shrink a KV cache

Serving systems shrink or share the cache in four ways: fewer KV heads, paged allocation, lower-precision storage, and offloading cold entries. Fewer KV heads is decided at training time, so you choose it by choosing a model. The other three are server settings you can test without changing the weights.

#Fewer KV heads

Multi-query attention shares one key-value head across all query heads. Grouped-query attention, from Ainslie et al., uses more than one but fewer than the query heads.

Llama 3.1 70B has 64 query heads and 8 KV heads. Its 320 KiB per token is eight times smaller than the 2,560 KiB full multi-head attention would need.

#Paged allocation

PagedAttention stores each sequence's cache in fixed-size blocks that need not be contiguous, and maps them through a table the way an operating system pages memory. Kwon et al. measured that existing systems used only 20.4 to 38.2 percent of their KV cache memory for actual token state.

#Quantized cache

Storing keys and values in 8 bits halves the cache. vLLM documents this as --kv-cache-dtype fp8 and warns that some attention types, such as sliding-window, are more sensitive to it. SGLang's --kv-cache-dtype accepts fp8_e5m2, fp8_e4m3, and 4-bit options.

#Offloading

Offloading moves inactive entries out of GPU memory. vLLM's cache config offers native and lmcache backends, and SGLang has --enable-hierarchical-cache. Offloading pays off when a prefix is reused more often than it is evicted, because each hit still costs a read from the slower tier.

The vLLM, SGLang, or llama.cpp comparison covers how these engines differ more broadly.

#How the KV cache relates to prompt caching

Prompt caching is a retained KV cache sold as a discount. The provider keeps the KV tensors for a prompt prefix, reuses them for later requests that match exactly, skips prefill for those tokens, and charges a reduced input rate. OpenAI's guide says this directly, and the other providers below document the same exact-prefix contract.

Inside one server, the same idea is prefix caching. vLLM hashes each cache block with its tokens and its parent's hash, and caches only full blocks. SGLang's RadixAttention keeps token prefixes in a radix tree and evicts least-recently-used leaves first.

Shared prefixes change the memory math. Take 32 users who each share a 16K-token system prompt and add 4K private tokens.

Without sharing, Llama 3.1 70B needs 655,360 cached tokens, or 200 GiB. With one shared prefix it needs 147,456 tokens, or 45 GiB, which is 4.4 times less.

Prices are from each provider's docs as of November 8, 2026. A 1.25x write and 0.1x read cost 67.5 percent of the uncached rate by the second request and 21.5 percent by the tenth. A 5-minute cache pays for itself on the first reuse.

A 2x write costs 105 percent on the second request and 73.3 percent on the third, so a 1-hour cache needs at least two reads.

Retention has a real memory cost.

A 50,000-token prefix on a Llama 3.1 70B-sized model occupies 15.3 GiB while it stays resident. That is a reasonable explanation for write premiums and lifetime limits, although no provider publishes the breakdown. Check Claude API pricing and Gemini API pricing for current per-token rates.

#How to budget a KV cache before you buy a GPU

Budget the cache by working backward from your p95 context length and target concurrency. The weights decide whether a model fits, and the cache decides how many users fit. Run these steps before you size a deployment, and repeat them whenever context limits or traffic change.

1. Run kv_size.py on your model's config.json and multiply by your p95 sequence length.
2. Multiply that by peak concurrent sequences, then add the weights in bytes at your serving precision.
3. Compare the total against 92 percent of your GPU memory.
4. If it does not fit, try a shared prefix first, then an FP8 cache, then a model with fewer KV heads or MLA, then more GPUs.

If the cache exceeds the weights at your target concurrency, spend on GPU memory before you spend on compute.

If you only call a hosted API, put stable content such as system prompts and tool definitions first, as the context engineering guide describes. Then confirm hits in usage.input_tokens_details.cached_tokens on OpenAI, cache_read_input_tokens on Anthropic, or prompt_cache_hit_tokens on DeepSeek.

More serving guides are on the inference topic page.

#Questions