Mixture of Experts (MoE)

In a dense model, every parameter runs on every input. A mixture-of-experts (MoE) layer splits part of the network into many expert sub-networks and adds a small router that picks a few experts for each token. The model can hold far more parameters than it uses on any one step. Noam Shazeer and colleagues made the idea work at scale in 2017 with a sparsely gated MoE layer of up to thousands of experts, reporting more than 1000x gains in model capacity with only minor losses in computational efficiency (Shazeer et al., Outrageously Large Neural Networks).

Adoption was slow at first. The Switch Transformer paper names complexity, communication costs and training instability as what held MoE back, and simplified routing so each token goes to a single expert; it reported up to 7x faster pre-training than comparable dense T5 models on the same compute and trained models with up to a trillion parameters (Fedus, Zoph and Shazeer, Switch Transformers). Open-weight MoE models followed. Mixtral 8x7B has eight expert blocks per layer and routes each token to two of them, so each token has access to 47B parameters but uses about 13B during inference (Jiang et al., Mixtral of Experts).

That last pair of numbers is the practical point for builders. Compute per token tracks the active parameters, but memory tracks the total: every expert has to be loaded, so Mixtral needs enough GPU memory to hold a dense 47B-parameter model (Hugging Face, Mixture of Experts Explained). So an MoE model runs faster than a dense model with the same total parameter count, but needs about as much memory as one, which shapes inference cost, hardware choice and whether quantization is needed. Routing also adds work of its own: if most tokens go to a few popular experts, training becomes inefficient, so MoE training adds mechanisms to balance the load. The model architecture topic covers the episodes on this layer, and can an open model cut your AI costs? covers the deployment side.