Architecture · Conditional computation
Mixture of Experts and top-k routing
In an ordinary network, all parameters work on every input. A Mixture of Experts (MoE) breaks that rule: a small gating network picks, for each input, a few specialised sub-networks and leaves the others idle3. The idea goes back to the 1990s1; its sparse version is the one found in large language models.
Experts
sub-networks of the same shape but with different parameters. In a Transformer, each one is a feed-forward network.
A router
A very light network that scores every expert from the input, then keeps only of them.
A balancing term
An auxiliary loss that pushes the router to spread the work; without it a few experts receive everything.
Anatomy of an MoE layer
Top-k routing with N = 4 experts and k = 2
- Routing decision
- Computation done by an active expert
- Idle expert: path not taken, no computation
- Auxiliary training signal
- Output of the layer
- Score every expert — The router multiplies the input by a learned matrix : one score per expert. During training, noise whose magnitude is itself learned is added.
- Keep only k experts — The top scores are kept, the others are set to . After the softmax their weight is exactly 0.
- Evaluate only the selected experts — The input is sent to experts 1 and 3 only. Experts 2 and 4 compute nothing: this is where the saving comes from.
- Combine the outputs — The output is the sum of the active experts' outputs, weighted by the router's weights.
- Watch the distribution — Over the whole batch, the load received by each expert is measured; an auxiliary loss penalises imbalance.
Sources:3
Score
One matrix multiplication yields scores. Its cost is negligible next to that of one expert.
Select
experts are kept, with . Some models route each token to a single expert5.
Compute
Only the selected experts run, each on the tokens it was assigned.
Recombine
The outputs are weighted and summed, then the next layer carries on as in an ordinary Transformer.
Routing in equations
Whenever there is no need to compute : the sparsity of the vector translates directly into computation avoided3.
| Symbol | Meaning | Unit |
|---|---|---|
| Input of the layer (one token's representation) | ||
| Expert , a neural network with its own parameters | ||
| Weight the router gives to expert ; the weights sum to 1 | ||
| Number of experts |
Sources:3
The noise, whose per-expert magnitude is learned, helps spread the load during training3.
| Symbol | Meaning | Unit |
|---|---|---|
| Weight matrix of the router | ||
| Learned matrix setting the noise magnitude per expert | ||
| Number of experts activated per input | ||
| Function , always positive |
Sources:3
Route tokens to experts
Synthetic data — seed 11For each token the gate computes a softmax over experts, keeps only the most probable and renormalises their weights; the output is the weighted sum of those experts. Simplification: gate logits are random draws (a fixed preference per expert plus a token-specific term) rather than the output of a trained router.
Interactive simulation: tokens are dispatched among several experts by top-k routing. It shows which experts are active for each token and the total load each one receives.
- Current token
- t8 (8 / 8)
- Selected experts
- E5, E3
- Gate mass kept
- 0.788
- Gate entropy (normalised)
- 0.525
- Final imbalance (max / mean)
- 2.5
Balancing the load
Share of tokens received by each expert
View the data
| Series | Expert | Share of tokens (%) |
|---|---|---|
| Without balancing | E1 | 41 |
| Without balancing | E2 | 3 |
| Without balancing | E3 | 27 |
| Without balancing | E4 | 2 |
| Without balancing | E5 | 12 |
| Without balancing | E6 | 1 |
| Without balancing | E7 | 9 |
| Without balancing | E8 | 5 |
| With balancing | E1 | 14 |
| With balancing | E2 | 12 |
| With balancing | E3 | 13 |
| With balancing | E4 | 11 |
| With balancing | E5 | 13 |
| With balancing | E6 | 12 |
| With balancing | E7 | 12 |
| With balancing | E8 | 13 |
The loss is zero when all experts receive the same total importance over the batch, and grows with the dispersion3.
| Symbol | Meaning | Unit |
|---|---|---|
| Batch of training examples | ||
| Coefficient of variation: standard deviation divided by the mean | ||
| Weight of the auxiliary loss, tuned by hand |
Sources:3
Under a uniform distribution, and the sum reaches its minimum . Since is not differentiable, the gradient flows through 5.
| Symbol | Meaning | Unit |
|---|---|---|
| Fraction of the batch's tokens dispatched to expert | ||
| Mean probability the router assigns to expert over the batch | ||
| Coefficient of the auxiliary loss |
Sources:5
Why it scales
Result derived in this sheet
Relation derived from the definition of the layer: adding experts raises , hence capacity, without touching , hence the computation per token. It ignores the cost of communication between machines.
| Symbol | Meaning | Unit |
|---|---|---|
| Number of parameters of one expert | ||
| Compute cost of one expert for one token | ||
| Cost of the router, small compared with |
| Criterion | Dense layer | Soft mixture1991 | Top-k MoEsparse |
|---|---|---|---|
| Experts evaluated per input | a single network, always | all | only |
| Compute per token | |||
| Parameters | |||
| Specific difficulty | none | cost grows with | balancing, non-differentiable selection |
Three ways of using sub-networks.
Dense layer
- Experts evaluated per input
- a single network, always
- Compute per token
- Parameters
- Specific difficulty
- none
Soft mixture1991
- Experts evaluated per input
- all
- Compute per token
- Parameters
- Specific difficulty
- cost grows with
Top-k MoEsparse
- Experts evaluated per input
- only
- Compute per token
- Parameters
- Specific difficulty
- balancing, non-differentiable selection
VerdictTop-k MoE keeps the capacity of the soft mixture at a compute cost close to that of a dense layer. This decoupling is what made it possible to train models with several hundred billion parameters4.
Check your understanding
QuestionAn MoE layer goes from 8 to 64 experts while keeping . What happens, roughly?
- AThe layer's parameters are multiplied by 8; computation per token barely changes.
- BBoth parameters and computation per token are multiplied by 8.
- CNothing changes, since only 2 experts are used.
Show the answer
- A — Correct. Correct: parameters follow , compute follows . Only the very light router computes 64 scores instead of 8.
- B — Incorrect. That would be true of a soft mixture, where every expert is evaluated.
- C — Incorrect. Two experts per token, but not always the same two: all 64 exist and must be stored and trained.
Are experts specialised by subject (medicine, law, code)?
Not by design. Nobody assigns roles to experts: the division of labour emerges from training, and routing is decided token by token from the internal representation. The word “expert” names a sub-network, not a human-readable skill.
How is this different from an ensemble of models?
An ensemble evaluates all models and averages their answers. An MoE is a single model, trained end to end, in which the router picks a few sub-networks inside one layer; the rest of the network is shared.
How does the router learn if top-k selection is not differentiable?
The gradient flows through the weights of the selected experts: those weights are a differentiable function of the scores. Discarded experts get no gradient for that example, hence the usefulness of noise and of the balancing loss so that each gets tried3.
References
- (1991). Adaptive Mixtures of Local Experts. Neural Computation, 3(1), 79–87. doi:10.1162/neco.1991.3.1.79
- (1994). Hierarchical Mixtures of Experts and the EM Algorithm. Neural Computation, 6(2), 181–214. doi:10.1162/neco.1994.6.2.181
- (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538. arxiv.org/abs/1701.06538
- (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668. arxiv.org/abs/2006.16668
- (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961. arxiv.org/abs/2101.03961