Explore modules

Mixture of Experts

Visualize how a gating network routes inputs to specialized expert networks to form a combined prediction.

Free theory article

Visualize how a gating network routes inputs to specialized expert networks to form a combined prediction.

Concept & interpretation

The Mixture of Experts (MoE) is an ensemble learning technique where multiple specialized neural networks, referred to as experts, collaborate to solve a problem. Instead of all parameters being used for every input, a separate gating network analyzes the input and selectively activates only a subset of the experts. This sparse activation allows the model to significantly increase its total parameter count and capacity without a proportional increase in the computational cost for a given prediction.

Read the full free theory article