Mixture of Experts
Visualize how a gating network routes inputs to specialized expert networks to form a combined prediction.
Free theory articleVisualize how a gating network routes inputs to specialized expert networks to form a combined prediction.
Concept & interpretation
The Mixture of Experts (MoE) is an ensemble learning technique where multiple specialized neural networks, referred to as experts, collaborate to solve a problem. Instead of all parameters being used for every input, a separate gating network analyzes the input and selectively activates only a subset of the experts. This sparse activation allows the model to significantly increase its total parameter count and capacity without a proportional increase in the computational cost for a given prediction.