---
title: "Transformer"
type: "glossary term"
url: "https://aifordevs.org/glossary#transformer"
markdown: "https://aifordevs.org/glossary/transformer.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#transformer"
---

# Transformer

A transformer is the neural network architecture behind modern LLMs, introduced in the 2017 paper Attention Is All You Need (Vaswani et al.). It stacks attention and feed-forward layers and processes all tokens of the input in parallel.

A transformer has no recurrence or convolution, so it adds positional encodings to the token embeddings to carry word order. Prefill can process a whole prompt at once because of this design, while generation still proceeds one token at a time.

## Example

The 2017 model used 512-dimensional layers and 8 attention heads. Llama 3.1 70B uses 80 layers, 8,192 dimensions, and 64 heads.

## Source

[Vaswani et al., Attention Is All You Need (2017)](https://arxiv.org/abs/1706.03762)

## Related terms

- [Attention](https://aifordevs.org/glossary#attention)
- [Mixture of experts](https://aifordevs.org/glossary#mixture-of-experts)

## Cite this term

AI for Devs. "Transformer." https://aifordevs.org/glossary#transformer

- Page: https://aifordevs.org/glossary#transformer
- Markdown: https://aifordevs.org/glossary/transformer.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs