A Survey on Efficient Architectures for LLMs

Analysis

Why This Paper Matters

Large Language Models (LLMs) have become foundational to modern AI, but their reliance on the transformer architecture introduces significant computational bottlenecks. This survey addresses a critical need by systematically cataloging architectural innovations that improve efficiency without sacrificing performance. As LLMs scale to hundreds of billions of parameters and are deployed in resource-constrained environments, understanding these alternatives is essential for both researchers and practitioners.

The paper is timely given the rapid proliferation of efficient architectures like mixture-of-experts (MoE) and linear attention, which are already being adopted in production systems. By grouping these approaches into a coherent taxonomy, the survey helps practitioners navigate a fragmented landscape and identify promising directions for their own work.

Technical Contributions

• Linear and Sparse Sequence Modeling: Covers methods that reduce the quadratic complexity of self-attention, such as linear attention (e.g., Performer, Linformer) and sparse attention patterns (e.g., Longformer, BigBird).
• Efficient Full Attention Variants: Includes techniques like FlashAttention and multi-query attention that optimize memory and compute without altering the attention mechanism's core.
• Sparse Mixture-of-Experts: Reviews MoE architectures (e.g., Switch Transformer, Mixtral) that activate only a subset of parameters per token, enabling massive model capacity with lower inference cost.
• Hybrid Models: Discusses combinations of the above techniques, such as integrating linear attention with MoE or convolutional layers, to leverage complementary strengths.
• Emerging Diffusion LLMs: Introduces diffusion-based language models as a new paradigm that may offer alternative efficiency trade-offs.

Results

As a survey, the paper does not present new experimental results. However, it synthesizes key findings from the literature: linear attention variants can reduce complexity from O(n²) to O(n) with minimal accuracy loss; MoE models can achieve up to 4x FLOPs reduction per token compared to dense models of equivalent quality; and hybrid architectures often outperform pure approaches on perplexity and throughput benchmarks. The survey notes that many efficient methods still face challenges in long-context scenarios and training stability.

Significance

This survey provides a valuable roadmap for the next generation of LLM architectures. By highlighting the trade-offs between efficiency and quality across different approaches, it empowers researchers to make informed design choices. The inclusion of cross-modal applications (e.g., vision, speech) broadens the impact beyond NLP. For practitioners at Neura Market, this work underscores the importance of architectural innovation in reducing deployment costs and enabling real-time applications. The survey also identifies open problems, such as scaling laws for non-transformer architectures and the need for standardized benchmarks, which will shape future research directions.