---
title: "Attention"
type: "glossary term"
url: "https://aifordevs.org/glossary#attention"
markdown: "https://aifordevs.org/glossary/attention.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#attention"
---

# Attention

Attention is the transformer operation that lets each token weigh the tokens before it when computing its own representation. Its cost grows with sequence length, which is why long contexts are expensive and why the KV cache exists.

Each token gets a query, a key, and a value vector. Scaled dot-product attention computes softmax(QK^T / sqrt(d_k)) V, so queries score keys and the scores weight the values. A model runs this across several heads in every layer.

## Example

A 1,000-token sequence with a causal mask has 500,500 query-key pairs per head, per layer.

## Source

[Vaswani et al., Attention Is All You Need (2017)](https://arxiv.org/abs/1706.03762)

## Related terms

- [Transformer](https://aifordevs.org/glossary#transformer)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [Grouped-query attention](https://aifordevs.org/glossary#grouped-query-attention)

## Read the article

[KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained)

## Cite this term

AI for Devs. "Attention." https://aifordevs.org/glossary#attention

- Page: https://aifordevs.org/glossary#attention
- Markdown: https://aifordevs.org/glossary/attention.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs