Twenty years of mixture of experts

Analysis

Why This Paper Matters

This paper provides a comprehensive survey of mixture of experts (ME) models, covering two decades of research. ME models are important for handling complex, heterogeneous data by dividing the input space among specialized expert networks, with a gating network controlling their contributions. The survey consolidates foundational models for regression and classification, as well as training techniques, making it a valuable resource for both newcomers and experienced researchers.

Technical Contributions

• Comprehensive survey: Covers fundamental ME models for regression and classification.
• Training discussion: Reviews algorithms for training ME models, including expectation-maximization and gradient-based methods.
• Historical perspective: Traces the evolution of ME over twenty years, highlighting key developments.

Results

As a survey, this paper does not present new experimental results. It synthesizes existing knowledge without providing quantitative comparisons or benchmarks.

Significance

This survey helps unify the understanding of mixture of experts, a technique with applications in various AI domains such as ensemble learning, multi-task learning, and hierarchical models. By providing a structured overview, it facilitates further research and practical adoption of ME models.