---
title: "Mixture of experts: how MoE works and what it costs"
type: "deep dive"
url: "https://aifordevs.org/articles/mixture-of-experts"
markdown: "https://aifordevs.org/articles/mixture-of-experts.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/mixture-of-experts"
description: "Mixture of experts LLMs load every expert but run only a few per token. Compare total and active parameters, and see what memory, batching, and routing cost."
author: "AI for Devs"
published: "2026-11-08"
level: "Advanced"
topics: ["models", "inference"]
---

# Mixture of experts: how MoE works and what it costs

Mixture of experts LLMs load every expert but run only a few per token. Compare total and active parameters, and see what memory, batching, and routing cost.

> **TL;DR**
>
> - Mixture of experts replaces a transformer's dense feed-forward layers with many parallel expert networks and a router that runs only a few of them per token.
> - Serving cost splits in two. Memory follows total parameters and compute follows active parameters, so DeepSeek-V3 loads 671B and computes 37B per token.
> - Batching removes the memory-bandwidth discount but keeps the compute discount. Under uniform routing, a Mixtral-shaped model touches 99 percent of its experts by batch 16.
> - As of November 8, 2026, model cards put most large open-weight MoE models at 3 to 6 percent active, compared with 27.7 percent for Mixtral 8x7B.
> - The occupancy script ran on Python 3.15.1 with a Monte Carlo cross-check. No MoE model or GPU ran for this post, so batching figures are arithmetic.

Mixture of experts (MoE) has two meanings, and search results for the term now treat the LLM meaning as the default. The classic meaning is the 1991 method of Jacobs, Jordan, Nowlan, and Hinton, in which a gating network assigns each training case to a few expert networks. A mixture-of-experts LLM is a [transformer](https://aifordevs.org/glossary#transformer) whose feed-forward layers are replaced by many parallel expert networks and a router that sends each token to only a few of them. The rest of this post covers that LLM design and what it costs an engineer who has to serve it.

## What mixture of experts means in an LLM

A [mixture of experts](https://aifordevs.org/glossary#mixture-of-experts) LLM keeps attention dense but replaces each feed-forward layer with several parallel feed-forward networks and a small router. The router scores every expert for each token and runs only the top few, so one token's compute touches a small fraction of the weights.

**Figure 1.** Attention runs fully for every token. The router scores all experts but only the top two compute, so this layer holds eight experts in memory and spends compute on two.

Routing happens per token and per layer. In [Mixtral 8x7B](https://arxiv.org/abs/2401.04088), each of 32 layers has 8 experts, a linear router picks the top two, and a softmax over those two scores weights their outputs. Only the selected experts run.

Some models add shared experts that every token uses. [DeepSeek-V3](https://arxiv.org/abs/2412.19437) has 1 shared expert and 256 routed experts per MoE layer, with 8 routed experts active per token, so each token goes through 9 experts per layer.

Experts replace only the feed-forward block, and [attention](https://aifordevs.org/glossary#attention) stays dense. [Fedus et al.](https://arxiv.org/abs/2101.03961) tested attention-layer experts and left them out because they were unstable in bfloat16. Dense attention also means an MoE model has the same [KV cache](https://aifordevs.org/articles/kv-cache-explained) cost as a dense model with the same attention layout.

## How MoE moved from 1991 to today's LLMs

Mixture of experts started as a 1991 way to split training cases across networks, became a sparse layer in 2017, and moved into large transformers in 2020 and 2021. Mixtral 8x7B then shipped as an open-weight model built on that design, and later large open-weight models followed.

| Year | Work | What changed |
| --- | --- | --- |
| 1991 | [Jacobs et al.](https://www.cs.toronto.edu/~fritz/absps/jjnh91.pdf) | A gating network assigns training cases to experts |
| 2017 | [Shazeer et al.](https://arxiv.org/abs/1701.06538) | Noisy top-k gating and balancing losses; up to 137B parameters and 131,072 experts in an LSTM |
| 2020 | [GShard](https://arxiv.org/abs/2006.16668) | Experts replace feed-forward layers in a transformer; 600B parameters with 2,048 experts per layer; top-2 gating |
| 2021 | [Switch Transformers](https://arxiv.org/abs/2101.03961) | Top-1 routing, 1.6T parameters, and author-reported pre-training speedups of up to 7x over T5 at equal compute |
| 2024 | [Mixtral 8x7B](https://arxiv.org/abs/2401.04088) | Open weights: 47B total and 13B active |
| 2024 | [DeepSeek-V3](https://arxiv.org/abs/2412.19437) | 671B total and 37B active, balanced by per-expert bias terms with only a small sequence-wise loss |

Scale is the through line. Shazeer et al. got up to 137B parameters into a 2017 LSTM model and reported best results at 68B. Switch reached 1.6T with one expert per token.

Later designs activate more experts per token, two in Mixtral and eight routed plus one shared in DeepSeek-V3.

## Why load balancing is the hard part of MoE

A router without balancing can concentrate tokens on a few experts, which wastes the others in training and overloads them in serving. Shazeer et al., Switch Transformers, and DeepSeek-V3 each added a mechanism to spread the load, from auxiliary losses and capacity limits to per-expert bias terms.

[Shazeer et al.](https://arxiv.org/abs/1701.06538) added an importance loss and a load loss that penalize uneven gate values across a batch. [Switch Transformers](https://arxiv.org/abs/2101.03961) used one auxiliary loss with weight 0.01, and set an expert capacity so overflow tokens pass through the residual connection. Dropped tokens were typically under 1 percent in those experiments.

[DeepSeek-V3](https://arxiv.org/abs/2412.19437) replaced the per-step auxiliary loss with a bias term per expert. After each step, overloaded experts' biases go down and underloaded ones go up, which shifts routing without distorting the gate values used for output. A small sequence-wise balance loss remains as a guard against extreme imbalance within one sequence.

Routing is not a topic switch. [Mixtral's authors](https://arxiv.org/abs/2401.04088) report no obvious topic-based pattern in expert assignment across their layers, with assignment following syntax more than domain.

Consecutive tokens often reused the same expert at higher layers. That locality matters for serving, because it can oversubscribe a device.

## Loaded versus active parameters in open-weight MoE models

Most large open-weight MoE models load 18 to 33 times more parameters than they run per token. Model cards put DeepSeek-V4-Pro at 1.6T total and 49B active, and Kimi K3 at 2.8T total and 104B active, as of November 8, 2026. Mixtral 8x7B's ratio was 3.6 times.

| Model | Total | Active per token | Active share | Total / active | Experts per layer |
| --- | --- | --- | --- | --- | --- |
| [Mixtral 8x7B](https://arxiv.org/abs/2401.04088) | 47B | 13B | 27.7% | 3.6x | 2 of 8 |
| [Llama 4 Scout](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) | 109B | 17B | 15.6% | 6.4x | 16 experts |
| [DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3) | 671B | 37B | 5.5% | 18.1x | 8 of 256 + 1 shared |
| [GLM-5](https://huggingface.co/zai-org/GLM-5) | 744B | 40B | 5.4% | 18.6x | 8 of 256 + 1 shared |
| [MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3) | 428B | 23B | 5.4% | 18.6x | 4 of 128 + 1 shared |
| [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) | 117B | 5.1B | 4.4% | 22.9x | 4 of 128 |
| [Llama 4 Maverick](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) | 400B | 17B | 4.3% | 23.5x | 128 experts |
| [Kimi K3](https://huggingface.co/moonshotai/Kimi-K3) | 2.8T | 104B | 3.7% | 26.9x | 16 of 896 |
| [Trinity-Large-Preview](https://huggingface.co/arcee-ai/Trinity-Large-Preview) | 398B | 13B | 3.3% | 30.6x | 4 of 256 (1 shared) |
| [Kimi K2](https://huggingface.co/moonshotai/Kimi-K2-Instruct) | 1T | 32B | 3.2% | 31.3x | 8 of 384 + 1 shared |
| [DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) | 1.6T | 49B | 3.1% | 32.7x | 6 of 384 + 1 shared |

Totals and active counts are as stated on each vendor's page as of November 8, 2026, sorted by active share, with ratios recomputed by code. Expert counts come from each model's Hugging Face `config.json` or card. Meta's post gives expert counts for Llama 4 but not how many activate per token.

A count from [DeepSeek-V3's config](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json) explains the split. Each expert holds about 44M parameters per layer, and 58 MoE layers with 257 experts come to 656.5B, or 97.8 percent of 671B. One token runs 9 experts per layer, which adds 23.0B to the remaining 14.5B and lands near the published 37B.

Hugging Face files can exceed the headline. [DeepSeek's README](https://github.com/deepseek-ai/DeepSeek-V3) notes 685B on disk, because 14B of multi-token prediction weights sit on top of the 671B main model. Those weights are what [speculative decoding](https://aifordevs.org/articles/speculative-decoding) can reuse as a draft.

## What mixture of experts costs in memory

MoE memory follows total parameters, because any token can route to any expert on the next step, so every expert stays in GPU memory. Mixtral 8x7B needs about 93 GB of 16-bit weights to run 13B active parameters, about 5.8 times the weights of Llama 3.1 8B.

[Mixtral's authors](https://arxiv.org/abs/2401.04088) state that memory cost for serving tracks the sparse parameter count, 47B. The weights do not shrink because routing is sparse, and routing cannot be predicted before the token exists.

Run the budget on a [141 GB H200](https://www.nvidia.com/en-us/data-center/h200/), using 92 percent as usable memory. Llama 3.1 8B leaves 105.9 GiB for the cache, which holds 105 sequences at 8K tokens. Mixtral 8x7B leaves 33.8 GiB after 93.4 GB of weights, which holds 33 sequences, because both models need 128 KiB of cache per token.

The largest models do not fit on one GPU. [DeepSeek-V3's published weights](https://huggingface.co/deepseek-ai/DeepSeek-V3) total about 688 GB, mostly 8-bit, or 5.3 H200s at 92 percent. [GLM-5](https://huggingface.co/zai-org/GLM-5) at 16 bits needs about 1.5 TB, or 11.6 H200s, before any cache.

Quantizing experts is a common escape. [gpt-oss-120b's card](https://huggingface.co/openai/gpt-oss-120b) says MXFP4 quantization of its MoE weights lets it fit a single 80 GB GPU, and [DeepSeek-V4-Pro's card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) lists FP4 precision for MoE expert parameters. Test [quantization](https://aifordevs.org/glossary#quantization) on your own task before you serve it.

## What mixture of experts costs in compute

MoE compute follows active parameters, so a Mixtral 8x7B token runs about 13B parameters and a DeepSeek-V3 token about 37B. That saving holds at every batch size. The price is routing overhead and extra memory loads when more than one expert runs per device, which the [Mixtral paper](https://arxiv.org/abs/2401.04088) names as a hardware efficiency cost.

A sparse layer does work a dense layer does not: scoring experts, then grouping tokens by expert before each feed-forward multiply. Mixtral's authors report that overhead, along with MegaBlocks kernels in vLLM that recast the expert multiplies as sparse matrix work. They say the design suits batched workloads, where each expert still sees enough tokens to use the hardware well.

Per-expert batches are small during decode and large during prefill. Mean tokens per expert per layer is `batch x k / experts`, so a DeepSeek-V3-shaped layer at decode batch 256 gives each expert about 8 tokens. A 10,000-token prefill gives each expert about 313 tokens, which is why [prefill and decode](https://aifordevs.org/glossary#prefill-and-decode) load the experts so differently.

DeepSeek reports the same regime at production scale. In its [V3 decoding deployment](https://arxiv.org/abs/2412.19437), the report says per-expert batches usually stay within 256 tokens and that the bottleneck is memory access rather than computation.

## How batching changes the mixture of experts math

At batch 1, a decode step reads only the active experts' weights. As batch size grows, tokens spread across experts, so a step soon reads most of them. Under uniform routing, a Mixtral-shaped model touches 99 percent of its experts by batch 16, which removes the memory-bandwidth discount and keeps the compute discount.

For Mixtral that is 13B of 47B at batch 1 and all 47B at batch 32, shared across 32 tokens. Expected occupancy has a closed form when routing is uniform.

```python title="moe_touch.py"
"""Share of routed experts a batch touches, assuming uniform routing."""


def share_touched(k: int, experts: int, batch: int) -> float:
    """Expected share of routed experts used by at least one token."""
    return 1 - (1 - k / experts) ** batch


models = [("Mixtral 8x7B", 2, 8), ("DeepSeek-V3", 8, 256), ("Kimi K3", 16, 896)]
batches = (1, 8, 16, 32, 64, 128, 256)
print(f"{'batch':<14}", *(f"{b:>6}" for b in batches))
for name, k, n in models:
    row = (f"{share_touched(k, n, b):6.1%}" for b in batches)
    print(f"{name:<14}", *row)
```

```text
batch               1      8     16     32     64    128    256
Mixtral 8x7B    25.0%  90.0%  99.0% 100.0% 100.0% 100.0% 100.0%
DeepSeek-V3      3.1%  22.4%  39.8%  63.8%  86.9%  98.3% 100.0%
Kimi K3          1.8%  13.4%  25.0%  43.8%  68.4%  90.0%  99.0%
```

A Monte Carlo run with random top-k routing matched three sampled entries to within 0.05 percentage points. The script ran on Python 3.15.1 and no model was involved. Real routers can be skewed or show temporal locality, and both reduce occupancy, so read these as upper bounds on how many experts a batch touches.

**Figure 2.** Under uniform routing, batch size decides how much of an MoE model each decode step reads. Mixtral's experts are 99 percent touched by batch 16, and DeepSeek-V3's by batch 128.

Mixtral's numbers show the trade. The table counts weights read in one step, using the same uniform routing assumption, against a dense model with 13B or 47B parameters.

| Batch | Experts touched | Mixtral weights read | Mixtral per token | Dense 13B per token | Dense 47B per token |
| --- | --- | --- | --- | --- | --- |
| 1 | 25.0% | 12.9B | 12.88B | 13.00B | 46.70B |
| 4 | 68.4% | 32.4B | 8.11B | 3.25B | 11.68B |
| 8 | 90.0% | 42.2B | 5.27B | 1.62B | 5.84B |
| 16 | 99.0% | 46.3B | 2.89B | 0.81B | 2.92B |
| 32 | 100.0% | 46.7B | 1.46B | 0.41B | 1.46B |

At batch 1, Mixtral reads no more than a dense 13B model. By batch 16, it reads as much per token as a dense 47B, and at batch 32 it spends 3.6 times the memory traffic per token of a dense 13B.

The compute discount is still there at batch 32, and the bandwidth discount is gone. If your deployment runs at [large batch sizes](https://aifordevs.org/glossary#continuous-batching), judge an MoE model on its compute and capacity numbers.

## How expert parallelism spreads experts across GPUs

Expert parallelism puts different experts on different GPUs and ships each token across the network to the experts it needs. vLLM documents this for MoE models, and DeepSeek's V3 decoding fleet ran 320-way expert parallelism. The cost is all-to-all traffic, and the slowest GPU sets each step's time.

**Figure 3.** Expert parallelism spreads experts across GPUs, so tokens cross the network to reach them. One busy expert, E5 here, sets the step time for its whole group, which is why load balancers duplicate hot experts.

Without [expert parallelism](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/), vLLM runs MoE layers with tensor parallelism across the group, like a dense model. With `--enable-expert-parallel`, vLLM shards experts across EP ranks, and the expert count per rank is the total experts divided by the rank count.

[DeepSeek's V3 report](https://arxiv.org/abs/2412.19437) gives a production layout. Its minimum decoding unit is 40 nodes with 320 GPUs, each GPU hosts one expert, and 64 GPUs host redundant and shared experts. The prefilling unit is 4 nodes with 32 GPUs using 32-way expert parallelism.

The communication has its own libraries. [DeepEP](https://github.com/deepseek-ai/DeepEP) provides the all-to-all dispatch and combine kernels that move tokens to and from experts.

Imbalance sets the ceiling, because a step waits for the busiest expert. [EPLB](https://github.com/deepseek-ai/EPLB), DeepSeek's balancer, duplicates hot experts and packs them across GPUs. vLLM's `--enable-eplb` moves experts based on load statistics and warns that redundant experts cost extra memory, so on a tight deployment the fix can break your memory budget.

## What to check before you serve an MoE model

Before you serve a mixture of experts model, price memory first, then decide whether expert parallelism is worth the network. If the weights and your KV cache fit on one GPU, run it as a single model and skip expert parallelism. Run these checks in order.

1. Multiply total parameters by bytes per parameter at your serving precision, using the model card and the Hugging Face file sizes.
2. Add the [KV cache](https://aifordevs.org/articles/kv-cache-explained) for your target concurrency and context length. Attention stays dense, so MoE does not change it.
3. Estimate decode batch size, then run `moe_touch.py` to see how much of the model each step reads.
4. If the weights exceed one GPU, plan the interconnect and load balancing before the GPU count, because the slowest expert sets the step time.

Skip a large MoE model for an interactive endpoint with a tight memory budget, because the weights will crowd out the cache. Choose it when you can batch traffic and want total-parameter capacity at active-parameter compute.

Check the [engine comparison](https://aifordevs.org/articles/vllm-sglang-llama-cpp) for your server choice and the [best LLM for coding](https://aifordevs.org/articles/best-llm-for-coding) comparison for which models are strong on code today. New open-weight MoE releases land often, so watch the [news page](https://aifordevs.org/news) and the [models topic page](https://aifordevs.org/topics/models), and read each model card before you size hardware. The [inference topic page](https://aifordevs.org/topics/inference) collects more serving guides.

## Questions

### What is mixture of experts in an LLM?

Mixture of experts is a transformer design where each feed-forward layer is replaced by several parallel networks, called experts, plus a router. For every token at every layer, the router scores the experts and runs only the top few. Attention stays dense. Models such as Mixtral 8x7B, DeepSeek-V3, and Kimi K3 use this design.

### What does active parameters mean for an MoE model?

Active parameters are the weights one token actually runs through. They equal the non-expert weights, mostly attention and normalization, plus the chosen experts per layer. Total parameters are every weight in the checkpoint. DeepSeek-V3 has 671B total and 37B active, so only about 5.5 percent of its weights do work for any given token.

### Why can't you load only the active experts?

The router picks experts from each token's own hidden state at each layer, so the required experts are only known at run time. Different tokens, and later layers for the same token, pick different experts. Paging an expert in from storage mid-step would cost far more than holding it in GPU memory.

### How much memory does an MoE model need?

MoE memory follows total parameters, and quantization lowers the bytes per parameter. Mixtral 8x7B needs about 93 GB of 16-bit weights. DeepSeek-V3 shipped as mostly 8-bit checkpoints at about 688 GB, and GLM-5 needs about 1.5 TB at 16 bits. Attention is unchanged by MoE, so KV cache needs stay separate and add on top.

### Is a mixture of experts model faster than a dense model?

An MoE model computes only its active parameters per token, so compute per token is close to a dense model of the active size. Per-token memory traffic depends on batch size. At batch 1 it matches the active size, and at large batches it approaches the total weight count, because almost every expert gets touched in each step.

### Do mixture of experts models specialize experts by topic?

Mostly no. The Mixtral paper reports no obvious topic-based assignment of experts across layers, and notes that routing tends to follow syntax more than domain. Consecutive tokens often reuse the same expert at higher layers, which the authors call temporal locality. Treat experts as a compute-saving structure, not as named specialists you can target.

## Sources

1. [Jacobs, Jordan, Nowlan, and Hinton, Adaptive Mixtures of Local Experts (1991)](https://www.cs.toronto.edu/~fritz/absps/jjnh91.pdf)
2. [Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017)](https://arxiv.org/abs/1701.06538)
3. [Lepikhin et al., GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020)](https://arxiv.org/abs/2006.16668)
4. [Fedus, Zoph, and Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021)](https://arxiv.org/abs/2101.03961)
5. [Jiang et al., Mixtral of Experts (2024)](https://arxiv.org/abs/2401.04088)
6. [DeepSeek-AI, DeepSeek-V3 Technical Report (2024), Sections 2.1, 3.4, and 4.2](https://arxiv.org/abs/2412.19437)
7. [DeepSeek-V3 README on GitHub, total parameter note](https://github.com/deepseek-ai/DeepSeek-V3)
8. [DeepSeek-V3 config.json on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json)
9. [Meta, The Llama 4 herd (2025)](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)
10. [OpenAI, gpt-oss-120b model card](https://huggingface.co/openai/gpt-oss-120b)
11. [Moonshot AI, Kimi K2 model card](https://huggingface.co/moonshotai/Kimi-K2-Instruct)
12. [Moonshot AI, Kimi K3 model card](https://huggingface.co/moonshotai/Kimi-K3)
13. [Z.ai, GLM-5 model card](https://huggingface.co/zai-org/GLM-5)
14. [MiniMax, MiniMax-M3 model card](https://huggingface.co/MiniMaxAI/MiniMax-M3)
15. [Arcee AI, Trinity-Large-Preview model card](https://huggingface.co/arcee-ai/Trinity-Large-Preview)
16. [DeepSeek-AI, DeepSeek-V4-Pro model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)
17. [Mixtral-8x7B model page on Hugging Face](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)
18. [NVIDIA H200 specifications](https://www.nvidia.com/en-us/data-center/h200/)
19. [vLLM docs, Expert Parallel Deployment](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/)
20. [DeepSeek, DeepEP](https://github.com/deepseek-ai/DeepEP)
21. [DeepSeek, EPLB](https://github.com/deepseek-ai/EPLB)

## Cite this deep dive

AI for Devs. "Mixture of experts: how MoE works and what it costs." Published November 8, 2026. https://aifordevs.org/articles/mixture-of-experts

- Page: https://aifordevs.org/articles/mixture-of-experts
- Markdown: https://aifordevs.org/articles/mixture-of-experts.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08