Aller au contenu

IAApprentissage profondExplicationAvancé

Par · Publiée le · 11 min de lecture

Sommaire
  1. Anatomy of an MoE layer
  2. Routing in equations
  3. Balancing the load
  4. Why it scales
  5. Check your understanding

Après cette fiche, vous saurez

  • Describe the role of the gating network and of the experts.
  • Write noisy top-k gating and explain what each term does.
  • Explain why a load-balancing loss is needed.
  • Separate the growth of parameters from the growth of computation per token.

Formulaire

Les équations de la fiche, d’un coup d’œil

  1. 01Output of the layery=∑i=1NG(x)i Ei(x)y = \sum_{i=1}^{N} G(x)_i \, E_i(x)
  2. 02Noisy top-k gatingG(x)=Softmax(KeepTopK(H(x),k))H(x)i=(x⋅Wg)i+εi⋅Softplus((x⋅Wnoise)i),εi∼N(0,1)KeepTopK(v,k)i={viif vi is among the top k elements of v−∞otherwise\begin{aligned} G(x) &= \mathrm{Softmax}\big(\mathrm{KeepTopK}(H(x), k)\big) \\ H(x)_i &= (x \cdot W_g)_i + \varepsilon_i \cdot \mathrm{Softplus}\big((x \cdot W_{\text{noise}})_i\big), \quad \varepsilon_i \sim \mathcal{N}(0, 1) \\ \mathrm{KeepTopK}(v, k)_i &= \begin{cases} v_i & \text{if } v_i \text{ is among the top } k \text{ elements of } v \\ -\infty & \text{otherwise} \end{cases} \end{aligned}
  3. 03Importance loss (Shazeer et al.)Importance(X)=∑x∈XG(x)Limportance(X)=wimportance⋅CV(Importance(X))2\begin{aligned} \mathrm{Importance}(X) &= \sum_{x \in X} G(x) \\ L_{\text{importance}}(X) &= w_{\text{importance}} \cdot \mathrm{CV}\big(\mathrm{Importance}(X)\big)^{2} \end{aligned}
  4. 04Auxiliary loss (Switch Transformer)Laux=α⋅N⋅∑i=1Nfi⋅PiL_{\text{aux}} = \alpha \cdot N \cdot \sum_{i=1}^{N} f_i \cdot P_i
  5. 05Parameters and compute decouplePMoE≈N⋅PE⏟parametersCtoken≈k⋅CE+Crouter⏟compute per token\underbrace{P_{\text{MoE}} \approx N \cdot P_E}_{\text{parameters}} \qquad \underbrace{C_{\text{token}} \approx k \cdot C_E + C_{\text{router}}}_{\text{compute per token}}

Demonstration example — not reviewed by the author

Architecture · Conditional computation

Mixture of Experts and top-k routing

In an ordinary network, all parameters work on every input. A Mixture of Experts (MoE) breaks that rule: a small gating network picks, for each input, a few specialised sub-networks and leaves the others idle3. The idea goes back to the 1990s1; its sparse version is the one found in large language models.

  • Experts

    NN sub-networks of the same shape but with different parameters. In a Transformer, each one is a feed-forward network.

  • A router

    A very light network that scores every expert from the input, then keeps only kk of them.

  • A balancing term

    An auxiliary loss that pushes the router to spread the work; without it a few experts receive everything.

Anatomy of an MoE layer

Top-k routing with N = 4 experts and k = 2

Diagram of a mixture-of-experts layer. The input vector goes to a router, which computes one score per expert, adds noise, keeps the two best scores and applies a softmax. Of four experts, two are active and receive the input, two are idle. The two outputs are summed with the router's weights to give the output. A dashed arrow links the router's weights to a load-balancing loss.ROUTEREXPERTSInput xone token's representationRouter scoresone per expertTunable noiseduring trainingKeep the top kSoftmaxweights G(x)Expert 1activeExpert 2idleExpert 3activeExpert 4idleWeighted sumOutput yBalancing lossadded to the main loss+the others: −∞E₁(x)E₃(x)weightsbatch statistics
  • Routing decision
  • Computation done by an active expert
  • Idle expert: path not taken, no computation
  • Auxiliary training signal
  • Output of the layer
  1. Score every expert — The router multiplies the input by a learned matrix WgW_g: one score per expert. During training, noise whose magnitude is itself learned is added.
  2. Keep only k experts — The top kk scores are kept, the others are set to −∞-\infty. After the softmax their weight is exactly 0.
  3. Evaluate only the selected experts — The input is sent to experts 1 and 3 only. Experts 2 and 4 compute nothing: this is where the saving comes from.
  4. Combine the outputs — The output is the sum of the active experts' outputs, weighted by the router's weights.
  5. Watch the distribution — Over the whole batch, the load received by each expert is measured; an auxiliary loss penalises imbalance.
Only the experts kept by the router are evaluated; their outputs are combined with the router's weights3.

Sources:3

  1. Score

    One matrix multiplication yields NN scores. Its cost is negligible next to that of one expert.

  2. Select

    kk experts are kept, with k≪Nk \ll N. Some models route each token to a single expert5.

  3. Compute

    Only the selected experts run, each on the tokens it was assigned.

  4. Recombine

    The outputs are weighted and summed, then the next layer carries on as in an ordinary Transformer.

Routing in equations

Output of the layer
y=∑i=1NG(x)i Ei(x)y = \sum_{i=1}^{N} G(x)_i \, E_i(x)

Whenever G(x)i=0G(x)_i = 0 there is no need to compute Ei(x)E_i(x): the sparsity of the vector G(x)G(x) translates directly into computation avoided3.

Variables
SymbolMeaningUnit
xxInput of the layer (one token's representation)
EiE_iExpert ii, a neural network with its own parameters
G(x)iG(x)_iWeight the router gives to expert ii; the weights sum to 1
NNNumber of experts

Sources:3

Noisy top-k gating
G(x)=Softmax(KeepTopK(H(x),k))H(x)i=(x⋅Wg)i+εi⋅Softplus((x⋅Wnoise)i),εi∼N(0,1)KeepTopK(v,k)i={viif vi is among the top k elements of v−∞otherwise\begin{aligned} G(x) &= \mathrm{Softmax}\big(\mathrm{KeepTopK}(H(x), k)\big) \\ H(x)_i &= (x \cdot W_g)_i + \varepsilon_i \cdot \mathrm{Softplus}\big((x \cdot W_{\text{noise}})_i\big), \quad \varepsilon_i \sim \mathcal{N}(0, 1) \\ \mathrm{KeepTopK}(v, k)_i &= \begin{cases} v_i & \text{if } v_i \text{ is among the top } k \text{ elements of } v \\ -\infty & \text{otherwise} \end{cases} \end{aligned}

The noise, whose per-expert magnitude is learned, helps spread the load during training3.

Variables
SymbolMeaningUnit
WgW_gWeight matrix of the router
WnoiseW_{\text{noise}}Learned matrix setting the noise magnitude per expert
kkNumber of experts activated per input
Softplus\mathrm{Softplus}Function z↦ln⁡(1+ez)z \mapsto \ln(1 + e^{z}), always positive

Sources:3

Route tokens to experts

Synthetic data — seed 11

For each token the gate computes a softmax over NN experts, keeps only the kk most probable and renormalises their weights; the output is the weighted sum of those kk experts. Simplification: gate logits are random draws (a fixed preference per expert plus a token-specific term) rather than the output of a trained router.

Interactive simulation: tokens are dispatched among several experts by top-k routing. It shows which experts are active for each token and the total load each one receives.

Tokenst1t2t3t4t5t6t7t8Gate: softmax(z / τ), then top-kE1p 0.06—E2p 0.02—E3p 0.07w 0.09E4p 0.07—E5p 0.72w 0.91E6p 0.03—E7p 0.02—E8p 0.02—Cumulative load: tokens routed to each expert3E10E24E32E45E52E60E70E8
Current token
t8 (8 / 8)
Selected experts
E5, E3
Gate mass kept
0.788
Gate entropy (normalised)
0.525
Final imbalance (max / mean)
2.5
Teaching simulation on randomly drawn scores (fixed seed): this is not the router of a trained model.

Balancing the load

Share of tokens received by each expert

Bar chart comparing, for eight experts, the share of tokens received without a balancing loss (two experts get the majority, several almost nothing) and with a balancing loss (shares close to 12.5% each).01020304050E1E2E3E4E5E6E7E8ExpertShare of tokens (%)Without balancingWith balancingILLUSTRATIVE DATA
Bar chart comparing, for eight experts, the share of tokens received without a balancing loss (two experts get the majority, several almost nothing) and with a balancing loss (shares close to 12.5% each).01020304050E1E3E5E7ExpertShare of tokens (%)Without balancingWith balancingILLUSTRATIVE DATA
Illustrative values chosen by hand to show the shape of the phenomenon; these are not measurements.
View the data
Bar chart comparing, for eight experts, the share of tokens received without a balancing loss (two experts get the majority, several almost nothing) and with a balancing loss (shares close to 12.5% each). — Illustrative data
SeriesExpertShare of tokens (%)
Without balancingE141
Without balancingE23
Without balancingE327
Without balancingE42
Without balancingE512
Without balancingE61
Without balancingE79
Without balancingE85
With balancingE114
With balancingE212
With balancingE313
With balancingE411
With balancingE513
With balancingE612
With balancingE712
With balancingE813
Importance loss (Shazeer et al.)
Importance(X)=∑x∈XG(x)Limportance(X)=wimportance⋅CV(Importance(X))2\begin{aligned} \mathrm{Importance}(X) &= \sum_{x \in X} G(x) \\ L_{\text{importance}}(X) &= w_{\text{importance}} \cdot \mathrm{CV}\big(\mathrm{Importance}(X)\big)^{2} \end{aligned}

The loss is zero when all experts receive the same total importance over the batch, and grows with the dispersion3.

Variables
SymbolMeaningUnit
XXBatch of training examples
CV\mathrm{CV}Coefficient of variation: standard deviation divided by the mean
wimportancew_{\text{importance}}Weight of the auxiliary loss, tuned by hand

Sources:3

Auxiliary loss (Switch Transformer)
Laux=α⋅N⋅∑i=1Nfi⋅PiL_{\text{aux}} = \alpha \cdot N \cdot \sum_{i=1}^{N} f_i \cdot P_i

Under a uniform distribution, fi=Pi=1/Nf_i = P_i = 1/N and the sum reaches its minimum 1/N1/N. Since fif_i is not differentiable, the gradient flows through PiP_i5.

Variables
SymbolMeaningUnit
fif_iFraction of the batch's tokens dispatched to expert ii
PiP_iMean probability the router assigns to expert ii over the batch
α\alphaCoefficient of the auxiliary loss

Sources:5

Why it scales

Parameters and compute decouple
PMoE≈N⋅PE⏟parametersCtoken≈k⋅CE+Crouter⏟compute per token\underbrace{P_{\text{MoE}} \approx N \cdot P_E}_{\text{parameters}} \qquad \underbrace{C_{\text{token}} \approx k \cdot C_E + C_{\text{router}}}_{\text{compute per token}}

Result derived in this sheet

Relation derived from the definition of the layer: adding experts raises NN, hence capacity, without touching kk, hence the computation per token. It ignores the cost of communication between machines.

Variables
SymbolMeaningUnit
PEP_ENumber of parameters of one expert
CEC_ECompute cost of one expert for one token
CrouterC_{\text{router}}Cost of the router, small compared with CEC_E
Three ways of using NN sub-networks.
CriterionDense layerSoft mixture1991Top-k MoEsparse
Experts evaluated per inputa single network, alwaysall NNonly kk
Compute per tokenCEC_EN⋅CEN \cdot C_Ek⋅CEk \cdot C_E
ParametersPEP_EN⋅PEN \cdot P_EN⋅PEN \cdot P_E
Specific difficultynonecost grows with NNbalancing, non-differentiable selection

Three ways of using NN sub-networks.

Dense layer

Experts evaluated per input
a single network, always
Compute per token
CEC_E
Parameters
PEP_E
Specific difficulty
none

Soft mixture1991

Experts evaluated per input
all NN
Compute per token
N⋅CEN \cdot C_E
Parameters
N⋅PEN \cdot P_E
Specific difficulty
cost grows with NN

Top-k MoEsparse

Experts evaluated per input
only kk
Compute per token
k⋅CEk \cdot C_E
Parameters
N⋅PEN \cdot P_E
Specific difficulty
balancing, non-differentiable selection

VerdictTop-k MoE keeps the capacity of the soft mixture at a compute cost close to that of a dense layer. This decoupling is what made it possible to train models with several hundred billion parameters4.

Check your understanding

QuestionAn MoE layer goes from 8 to 64 experts while keeping k=2k = 2. What happens, roughly?

  1. AThe layer's parameters are multiplied by 8; computation per token barely changes.
  2. BBoth parameters and computation per token are multiplied by 8.
  3. CNothing changes, since only 2 experts are used.
Show the answer
  • A — Correct. Correct: parameters follow NN, compute follows kk. Only the very light router computes 64 scores instead of 8.
  • B — Incorrect. That would be true of a soft mixture, where every expert is evaluated.
  • C — Incorrect. Two experts per token, but not always the same two: all 64 exist and must be stored and trained.
Are experts specialised by subject (medicine, law, code)?

Not by design. Nobody assigns roles to experts: the division of labour emerges from training, and routing is decided token by token from the internal representation. The word “expert” names a sub-network, not a human-readable skill.

How is this different from an ensemble of models?

An ensemble evaluates all models and averages their answers. An MoE is a single model, trained end to end, in which the router picks a few sub-networks inside one layer; the rest of the network is shared.

How does the router learn if top-k selection is not differentiable?

The gradient flows through the weights of the selected experts: those weights are a differentiable function of the scores. Discarded experts get no gradient for that example, hence the usefulness of noise and of the balancing loss so that each gets tried3.

Where does the idea come from?

In 1991 Jacobs, Jordan, Nowlan and Hinton proposed a system of experts made to compete by a gating network, which learns to split a task into sub-tasks1. Jordan and Jacobs extended it in 1994 to a hierarchy of experts2. The large-scale sparse version dates from 20173.

References

  1. Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, Geoffrey E. Hinton (1991). Adaptive Mixtures of Local Experts. Neural Computation, 3(1), 79–87. doi:10.1162/neco.1991.3.1.79
  2. Michael I. Jordan, Robert A. Jacobs (1994). Hierarchical Mixtures of Experts and the EM Algorithm. Neural Computation, 6(2), 181–214. doi:10.1162/neco.1994.6.2.181
  3. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538. arxiv.org/abs/1701.06538
  4. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668. arxiv.org/abs/2006.16668
  5. William Fedus, Barret Zoph, Noam Shazeer (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961. arxiv.org/abs/2101.03961

Aussi disponible en français

Partager cette fiche

Mots-clés

En vedetteIARecherche d’information

Le pipeline RAG, de l’indexation à la réponse

La génération augmentée par récupération donne à un modèle de langage des passages pertinents au moment de répondre. Sa qualité se joue d’abord à la récupération : ce qui n’est pas retrouvé ne peut pas être cité.

Intermédiaire · 11 min de lecture ·

En vedetteIAInformation RetrievalEN

The RAG pipeline, from indexing to answer

Retrieval-augmented generation hands a language model relevant passages at answer time. Its quality is decided first at retrieval: what is not found cannot be cited.

Intermédiaire · 10 min de lecture ·

En vedetteÉconométrieFinance

Le modèle GARCH(1,1) et la dynamique de la volatilité

Comment une récurrence à trois paramètres reproduit les grappes de volatilité des rendements financiers, à quelles conditions elle est stationnaire et comment on l’estime par quasi-maximum de vraisemblance.

Intermédiaire · 10 min de lecture ·