Preprint2024
Gemini 1.5 Pro
Unknown
Gemini 1.5 Pro is a compute-efficient multimodal mixture-of-experts model excelling in long-context retrieval and understanding across text, video, and audio.
0Feb 1, 2024Retrieval Augmented GenerationMultimodal
AI Research
AI & ML Research Papers
A comprehensive index of artificial intelligence and machine-learning research with AI-generated summaries, citation metrics, and direct links to papers and code.
All Papers
7 papersPreprint2024
MoE-LLaVA
Unknown
MoE-LLaVA introduces a sparse mixture-of-experts framework for vision-language models, activating only top-k experts per token to match larger model performance with fewer parameters.
0Jan 1, 2024
arXiv
Preprint2024
Mixtral 8x7B
Unknown
Introduces Mixtral 8x7B, a sparse mixture-of-experts language model trained on multilingual data with a 32k-token context window.
0Jan 1, 2024
Preprint2024
Fusemoe: Mixture-of-experts transformers for fleximodal fusion
Unknown
FuseMoE introduces a mixture-of-experts framework with a novel gating function for integrating diverse numbers of modalities.
0Jan 1, 2024Transformers
Preprint2025
Vimoe: An empirical study of designing vision mixture-of-experts
Unknown
This paper empirically studies the design of vision Mixture-of-Experts (MoE) models, providing insights into routing strategies and expert architectures for improved scalability.
0Jan 1, 2025Computer Vision
PreprintNeural Information Processing Systems2022
Towards understanding the mixture-of-experts layer in deep learning
Zixiang Chen, Yihe Deng, Yue Wu, et al.
This paper formally proves that Mixture-of-Experts layers provably improve test accuracy over a single expert in two-layer CNNs.
183Jan 1, 2022
PreprintarXiv.org2025
A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications
Siyuan Mu, Sen-Fon Lin
A comprehensive survey of Mixture-of-Experts architectures covering algorithms, theory, and applications.
115Mar 1, 2025Attention Mechanisms
arXiv