Examples
Worked examples
- Is an instance
Mixtral routes each token to 2 of 8 available experts per layer, so its active parameter count per token is far smaller than its total stored parameter count.
- Is an instance
Switch Transformer (2022) simplified MoE routing to a single top-1 expert per token to reduce routing complexity at scale.
Counter-examples
Looks similar, but isn't
- Not an instance
A dense transformer, where every parameter participates in processing every token, is not a mixture-of-experts model even if it is very large.
Editorial commentary
A mixture-of-experts (MoE) model replaces a single dense feed-forward block with many parallel ‘expert’ sub-networks and a learned router (gating network) that sends each input token to only a small subset of experts — typically the top one or two, out of dozens or hundreds. This decouples total parameter count from per-token compute cost: the model can hold far more parameters than a dense model of equivalent inference cost, because most of those parameters sit idle for any given token.
Total parameters vs active parameters
Because only a fraction of an MoE model’s experts fire per token, reporting practice distinguishes ‘total parameters’ (everything stored) from ‘active parameters per token’ (what actually does the compute for a given forward pass). A model described as having tens of billions of total parameters may run at the inference cost of a dense model a fraction that size — comparing MoE and dense models on total parameter count alone is misleading, since it says nothing about the compute or memory bandwidth actually spent per token.
Routing and training considerations
The router’s job — deciding which experts handle which tokens — creates its own engineering problems: without correction, routing tends to collapse onto a small number of favoured experts, under-using the rest of the model’s capacity. Training typically adds an auxiliary load-balancing loss to spread tokens more evenly across experts. Serving an MoE model also has different infrastructure demands than a dense model of the same total size, since all experts generally need to be resident in memory even though only a few are used per token, which shapes how these models get partitioned across hardware (expert parallelism).
Why it matters for frontier models
Shazeer et al. (2017) reintroduced sparse MoE for deep learning after earlier work in the 1990s; Fedus, Zoph and Shazeer’s Switch Transformer (2022) simplified routing to a single top-1 expert per token at scale. MoE architectures are now central to how several frontier models are built (publicly documented examples include Mixtral and GLaM, alongside architectures widely reported to use MoE without full public disclosure), because they offer a way to keep scaling capacity without a proportional increase in serving cost — which is also why ‘parameter count’ alone has become a less reliable proxy for a model’s real compute footprint than it once was.
References
- Shazeer et al., ‘Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer’ (ICLR, 2017); Fedus, Zoph, Shazeer, ‘Switch Transformer’ (JMLR, 2022).
Also known as
MoE · sparse mixture-of-experts
Machine-readable encodings
Use in your systems
<role vocab="credit"
vocab-identifier="https://casrai.org/dictionary/"
vocab-term="Mixture-of-experts (MoE)"
vocab-term-identifier="https://casrai.org/dictionary/term/mixture-of-experts-moe" />{
"@context": "https://schema.org",
"@type": "DefinedTerm",
"@id": "https://casrai.org/dictionary/term/mixture-of-experts-moe",
"name": "Mixture-of-experts (MoE)",
"identifier": "https://casrai.org/dictionary/term/mixture-of-experts-moe",
"description": "A model architecture that replaces a single dense layer with many parallel expert sub-networks and a learned router that sends each input to only a small subset of experts, decoupling total parameter count from per-token compute cost.",
"inDefinedTermSet": "https://casrai.org/dictionary/domain/ai-ml-research-outputs#set",
"url": "https://casrai.org/dictionary/term/mixture-of-experts-moe",
"sameAs": [
"MoE",
"sparse mixture-of-experts"
],
"license": "https://creativecommons.org/licenses/by/4.0/",
"publisher": {
"@id": "https://casrai.org/#organization"
},
"author": {
"@id": "https://casrai.org/#editorial-team"
},
"datePublished": "2026-05-21T02:22:51",
"dateModified": "2026-08-22T16:26:25",
"inLanguage": "en-GB",
"isAccessibleForFree": true
}
