---
title: "Grouped-query attention"
type: "glossary term"
url: "https://aifordevs.org/glossary#grouped-query-attention"
markdown: "https://aifordevs.org/glossary/grouped-query-attention.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#grouped-query-attention"
also_known_as: "GQA"
---

# Grouped-query attention (GQA)

Grouped-query attention (GQA) is an attention variant in which several query heads share one key and value head. It uses more key-value heads than multi-query attention and fewer than multi-head attention.

The KV cache stores only key and value heads, so cutting them shrinks cache memory and the memory read on every decode step. Ainslie et al. (2023) report that GQA reaches quality close to multi-head attention at a speed comparable to multi-query attention.

## Example

Llama 3.1 70B has 64 query heads and 8 key-value heads, so its KV cache is 8 times smaller than it would be with full multi-head attention.

## Source

[Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023)](https://arxiv.org/abs/2305.13245)

## Related terms

- [Attention](https://aifordevs.org/glossary#attention)
- [KV cache](https://aifordevs.org/glossary#kv-cache)

## Read the article

[KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained)

## Cite this term

AI for Devs. "Grouped-query attention." https://aifordevs.org/glossary#grouped-query-attention

- Page: https://aifordevs.org/glossary#grouped-query-attention
- Markdown: https://aifordevs.org/glossary/grouped-query-attention.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs