Mixture of experts: how MoE works and what it costs

Mixture of experts (MoE) has two meanings, and search results for the term now treat the LLM meaning as the default. The classic meaning is the 1991 method of Jacobs, Jordan, Nowlan, and Hinton, in which a gating network assigns each training case to a few expert networks. A mixture-of-experts LLM is a transformer whose feed-forward layers are replaced by many parallel expert networks and a router that sends each token to only a few of them. The rest of this post covers that LLM design and what it costs an engineer who has to serve it.

#What mixture of experts means in an LLM

A mixture of experts LLM keeps attention dense but replaces each feed-forward layer with several parallel feed-forward networks and a small router. The router scores every expert for each token and runs only the top few, so one token's compute touches a small fraction of the weights.

Routing happens per token and per layer. In Mixtral 8x7B, each of 32 layers has 8 experts, a linear router picks the top two, and a softmax over those two scores weights their outputs. Only the selected experts run.

Some models add shared experts that every token uses. DeepSeek-V3 has 1 shared expert and 256 routed experts per MoE layer, with 8 routed experts active per token, so each token goes through 9 experts per layer.

Experts replace only the feed-forward block, and attention stays dense. Fedus et al. tested attention-layer experts and left them out because they were unstable in bfloat16. Dense attention also means an MoE model has the same KV cache cost as a dense model with the same attention layout.

#How MoE moved from 1991 to today's LLMs

Mixture of experts started as a 1991 way to split training cases across networks, became a sparse layer in 2017, and moved into large transformers in 2020 and 2021. Mixtral 8x7B then shipped as an open-weight model built on that design, and later large open-weight models followed.

Scale is the through line. Shazeer et al. got up to 137B parameters into a 2017 LSTM model and reported best results at 68B. Switch reached 1.6T with one expert per token.

Later designs activate more experts per token, two in Mixtral and eight routed plus one shared in DeepSeek-V3.

#Why load balancing is the hard part of MoE

A router without balancing can concentrate tokens on a few experts, which wastes the others in training and overloads them in serving. Shazeer et al., Switch Transformers, and DeepSeek-V3 each added a mechanism to spread the load, from auxiliary losses and capacity limits to per-expert bias terms.

Shazeer et al. added an importance loss and a load loss that penalize uneven gate values across a batch. Switch Transformers used one auxiliary loss with weight 0.01, and set an expert capacity so overflow tokens pass through the residual connection. Dropped tokens were typically under 1 percent in those experiments.

DeepSeek-V3 replaced the per-step auxiliary loss with a bias term per expert. After each step, overloaded experts' biases go down and underloaded ones go up, which shifts routing without distorting the gate values used for output. A small sequence-wise balance loss remains as a guard against extreme imbalance within one sequence.

Routing is not a topic switch. Mixtral's authors report no obvious topic-based pattern in expert assignment across their layers, with assignment following syntax more than domain.

Consecutive tokens often reused the same expert at higher layers. That locality matters for serving, because it can oversubscribe a device.

#Loaded versus active parameters in open-weight MoE models

Most large open-weight MoE models load 18 to 33 times more parameters than they run per token. Model cards put DeepSeek-V4-Pro at 1.6T total and 49B active, and Kimi K3 at 2.8T total and 104B active, as of November 8, 2026. Mixtral 8x7B's ratio was 3.6 times.

Totals and active counts are as stated on each vendor's page as of November 8, 2026, sorted by active share, with ratios recomputed by code. Expert counts come from each model's Hugging Face config.json or card. Meta's post gives expert counts for Llama 4 but not how many activate per token.

A count from DeepSeek-V3's config explains the split. Each expert holds about 44M parameters per layer, and 58 MoE layers with 257 experts come to 656.5B, or 97.8 percent of 671B. One token runs 9 experts per layer, which adds 23.0B to the remaining 14.5B and lands near the published 37B.

Hugging Face files can exceed the headline. DeepSeek's README notes 685B on disk, because 14B of multi-token prediction weights sit on top of the 671B main model. Those weights are what speculative decoding can reuse as a draft.

#What mixture of experts costs in memory

MoE memory follows total parameters, because any token can route to any expert on the next step, so every expert stays in GPU memory. Mixtral 8x7B needs about 93 GB of 16-bit weights to run 13B active parameters, about 5.8 times the weights of Llama 3.1 8B.

Mixtral's authors state that memory cost for serving tracks the sparse parameter count, 47B. The weights do not shrink because routing is sparse, and routing cannot be predicted before the token exists.

Run the budget on a 141 GB H200, using 92 percent as usable memory. Llama 3.1 8B leaves 105.9 GiB for the cache, which holds 105 sequences at 8K tokens. Mixtral 8x7B leaves 33.8 GiB after 93.4 GB of weights, which holds 33 sequences, because both models need 128 KiB of cache per token.

The largest models do not fit on one GPU. DeepSeek-V3's published weights total about 688 GB, mostly 8-bit, or 5.3 H200s at 92 percent. GLM-5 at 16 bits needs about 1.5 TB, or 11.6 H200s, before any cache.

Quantizing experts is a common escape. gpt-oss-120b's card says MXFP4 quantization of its MoE weights lets it fit a single 80 GB GPU, and DeepSeek-V4-Pro's card lists FP4 precision for MoE expert parameters. Test quantization on your own task before you serve it.

#What mixture of experts costs in compute

MoE compute follows active parameters, so a Mixtral 8x7B token runs about 13B parameters and a DeepSeek-V3 token about 37B. That saving holds at every batch size. The price is routing overhead and extra memory loads when more than one expert runs per device, which the Mixtral paper names as a hardware efficiency cost.

A sparse layer does work a dense layer does not: scoring experts, then grouping tokens by expert before each feed-forward multiply. Mixtral's authors report that overhead, along with MegaBlocks kernels in vLLM that recast the expert multiplies as sparse matrix work. They say the design suits batched workloads, where each expert still sees enough tokens to use the hardware well.

Per-expert batches are small during decode and large during prefill. Mean tokens per expert per layer is batch x k / experts, so a DeepSeek-V3-shaped layer at decode batch 256 gives each expert about 8 tokens. A 10,000-token prefill gives each expert about 313 tokens, which is why prefill and decode load the experts so differently.

DeepSeek reports the same regime at production scale. In its V3 decoding deployment, the report says per-expert batches usually stay within 256 tokens and that the bottleneck is memory access rather than computation.

#How batching changes the mixture of experts math

At batch 1, a decode step reads only the active experts' weights. As batch size grows, tokens spread across experts, so a step soon reads most of them. Under uniform routing, a Mixtral-shaped model touches 99 percent of its experts by batch 16, which removes the memory-bandwidth discount and keeps the compute discount.

For Mixtral that is 13B of 47B at batch 1 and all 47B at batch 32, shared across 32 tokens. Expected occupancy has a closed form when routing is uniform.

"""Share of routed experts a batch touches, assuming uniform routing."""
 
 
def share_touched(k: int, experts: int, batch: int) -> float:
    """Expected share of routed experts used by at least one token."""
    return 1 - (1 - k / experts) ** batch
 
 
models = [("Mixtral 8x7B", 2, 8), ("DeepSeek-V3", 8, 256), ("Kimi K3", 16, 896)]
batches = (1, 8, 16, 32, 64, 128, 256)
print(f"{'batch':<14}", *(f"{b:>6}" for b in batches))
for name, k, n in models:
    row = (f"{share_touched(k, n, b):6.1%}" for b in batches)
    print(f"{name:<14}", *row)

batch               1      8     16     32     64    128    256
Mixtral 8x7B    25.0%  90.0%  99.0% 100.0% 100.0% 100.0% 100.0%
DeepSeek-V3      3.1%  22.4%  39.8%  63.8%  86.9%  98.3% 100.0%
Kimi K3          1.8%  13.4%  25.0%  43.8%  68.4%  90.0%  99.0%

A Monte Carlo run with random top-k routing matched three sampled entries to within 0.05 percentage points. The script ran on Python 3.15.1 and no model was involved. Real routers can be skewed or show temporal locality, and both reduce occupancy, so read these as upper bounds on how many experts a batch touches.

Mixtral's numbers show the trade. The table counts weights read in one step, using the same uniform routing assumption, against a dense model with 13B or 47B parameters.

At batch 1, Mixtral reads no more than a dense 13B model. By batch 16, it reads as much per token as a dense 47B, and at batch 32 it spends 3.6 times the memory traffic per token of a dense 13B.

The compute discount is still there at batch 32, and the bandwidth discount is gone. If your deployment runs at large batch sizes, judge an MoE model on its compute and capacity numbers.

#How expert parallelism spreads experts across GPUs

Expert parallelism puts different experts on different GPUs and ships each token across the network to the experts it needs. vLLM documents this for MoE models, and DeepSeek's V3 decoding fleet ran 320-way expert parallelism. The cost is all-to-all traffic, and the slowest GPU sets each step's time.

Without expert parallelism, vLLM runs MoE layers with tensor parallelism across the group, like a dense model. With --enable-expert-parallel, vLLM shards experts across EP ranks, and the expert count per rank is the total experts divided by the rank count.

DeepSeek's V3 report gives a production layout. Its minimum decoding unit is 40 nodes with 320 GPUs, each GPU hosts one expert, and 64 GPUs host redundant and shared experts. The prefilling unit is 4 nodes with 32 GPUs using 32-way expert parallelism.

The communication has its own libraries. DeepEP provides the all-to-all dispatch and combine kernels that move tokens to and from experts.

Imbalance sets the ceiling, because a step waits for the busiest expert. EPLB, DeepSeek's balancer, duplicates hot experts and packs them across GPUs. vLLM's --enable-eplb moves experts based on load statistics and warns that redundant experts cost extra memory, so on a tight deployment the fix can break your memory budget.

#What to check before you serve an MoE model

Before you serve a mixture of experts model, price memory first, then decide whether expert parallelism is worth the network. If the weights and your KV cache fit on one GPU, run it as a single model and skip expert parallelism. Run these checks in order.

1. Multiply total parameters by bytes per parameter at your serving precision, using the model card and the Hugging Face file sizes.
2. Add the KV cache for your target concurrency and context length. Attention stays dense, so MoE does not change it.
3. Estimate decode batch size, then run moe_touch.py to see how much of the model each step reads.
4. If the weights exceed one GPU, plan the interconnect and load balancing before the GPU count, because the slowest expert sets the step time.

Skip a large MoE model for an interactive endpoint with a tight memory budget, because the weights will crowd out the cache. Choose it when you can batch traffic and want total-parameter capacity at active-parameter compute.

Check the engine comparison for your server choice and the best LLM for coding comparison for which models are strong on code today. New open-weight MoE releases land often, so watch the news page and the models topic page, and read each model card before you size hardware. The inference topic page collects more serving guides.

#Questions