A survey on mixture of experts

Analysis

Why This Paper Matters

As AI models grow to unprecedented sizes, the computational cost of training and inference becomes a critical bottleneck. Mixture of experts (MoE) offers a promising solution by activating only a subset of parameters per input, enabling massive model capacity without proportional compute. This survey arrives at a pivotal moment when the AI community is actively seeking efficient scaling strategies. It consolidates a rapidly growing body of work, making it easier for practitioners to understand the landscape and choose appropriate MoE designs.

The paper's timing is particularly relevant given the rise of large language models and multimodal systems, where MoE has been successfully deployed (e.g., in models like Mixtral 8x7B). By providing a structured taxonomy, the survey helps demystify the various routing mechanisms, expert architectures, and training techniques that have been proposed.

Technical Contributions

The survey categorizes MoE approaches along several key dimensions:

• Architecture: Distinguishes between dense and sparse MoE, and discusses expert configurations (e.g., number of experts, capacity factor).
• Routing Mechanisms: Compares top-k routing, expert choice routing, and learned routing, highlighting their impact on load balancing and expert specialization.
• Training Strategies: Covers techniques such as auxiliary load balancing losses, expert dropout, and dynamic expert allocation.
• Applications: Reviews MoE in NLP, computer vision, and multi-task learning, showing domain-specific adaptations.

Results

While the survey does not present new experimental results, it aggregates key findings from the literature. For instance, MoE models can achieve comparable or better performance than dense models with 2-4x fewer FLOPs during inference. The paper notes that top-2 routing with a moderate number of experts (e.g., 8-64) provides a good balance between quality and efficiency. Load balancing remains a critical challenge, with auxiliary losses being the most common mitigation.

Significance

This survey provides a foundational reference for anyone working on efficient deep learning. By systematically organizing the MoE literature, it accelerates research and adoption of these techniques. The identified challenges—such as training instability, expert collapse, and inference latency—point to concrete areas for future work. As the demand for larger models continues, MoE will likely become a standard component in the AI engineer's toolkit.