---
title: "Distillation"
type: "glossary term"
url: "https://aifordevs.org/glossary#distillation"
markdown: "https://aifordevs.org/glossary/distillation.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#distillation"
---

# Distillation

Distillation is training a smaller student model to reproduce the outputs of a larger teacher model. It is a common way to get a cheaper model that keeps much of the teacher's quality on a narrow task.

Hinton et al. (2015) train the student on soft targets: both models divide their logits by a temperature above 1, so the teacher's near-miss answers carry signal, and the student returns to a temperature of 1 after training. DeepSeek's R1 paper reports that reasoning patterns from large models can also improve smaller ones.

## Example

DistilBERT is 40% smaller and 60% faster than BERT, and its authors report that it keeps 97% of BERT's language understanding performance.

## Source

[Hinton, Vinyals, and Dean, Distilling the Knowledge in a Neural Network (2015)](https://arxiv.org/abs/1503.02531)

## Related terms

- [Fine-tuning](https://aifordevs.org/glossary#fine-tuning)
- [Small language model](https://aifordevs.org/glossary#small-language-model)

## Cite this term

AI for Devs. "Distillation." https://aifordevs.org/glossary#distillation

- Page: https://aifordevs.org/glossary#distillation
- Markdown: https://aifordevs.org/glossary/distillation.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs