---
title: "Mixture of experts"
type: "glossary term"
url: "https://aifordevs.org/glossary#mixture-of-experts"
markdown: "https://aifordevs.org/glossary/mixture-of-experts.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#mixture-of-experts"
also_known_as: "MoE"
---

# Mixture of experts (MoE)

Mixture of experts (MoE) is a model architecture that replaces selected feed-forward layers with many expert sub-networks and routes each token to only a few of them. Total parameters can be very large while the compute per token stays close to that of a much smaller dense model.

Shazeer et al. (2017) introduced the sparsely gated MoE layer, where a trainable gating network selects experts for each input. Memory scales with total parameters because every expert must be loaded, while compute per token scales with the active ones.

## Example

DeepSeek-V3 has 671B total parameters and 37B active per token, about 5.5%. Mixtral 8x7B routes each token to 2 of 8 experts per layer, with about 13B of its 47B parameters active.

## Source

[Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017)](https://arxiv.org/abs/1701.06538)

## Related terms

- [Transformer](https://aifordevs.org/glossary#transformer)

## Read the article

[Mixture of experts: how MoE works and what it costs](https://aifordevs.org/articles/mixture-of-experts)

## Cite this term

AI for Devs. "Mixture of experts." https://aifordevs.org/glossary#mixture-of-experts

- Page: https://aifordevs.org/glossary#mixture-of-experts
- Markdown: https://aifordevs.org/glossary/mixture-of-experts.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs