Preprint
Large Language Models

EfficientLLM

Zhengqing Yuan, Weixiang Sun, Yixin Liu, Huichi Zhou, Rong Zhou, Yiyang Li, Zheyuan Zhang, Wei Song, Yue Huang, Haolong Jia, K. Murugesan, Yu Wang, Lifang He, Jianfeng Gao, Lichao Sun, Yanfang Ye
May 20, 2025arXiv.org4 citations
arXivPDFDOI

4

Citations

1

Influential Citations

arXiv.org

Venue

2025

Year

Abstract

Large Language Models (LLMs) have driven significant progress, yet their growing parameter counts and context windows incur prohibitive compute, energy, and monetary costs. We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale. Conducted on a production-class cluster (48xGH200, 8xH200 GPUs), our study systematically explores three key axes: (1) architecture pretraining (efficient attention variants: MQA, GQA, MLA, NSA; sparse Mixture-of-Experts (MoE)), (2) fine-tuning (parameter-efficient methods: LoRA, RSLoRA, DoRA), and (3) inference (quantization methods: int4, float16). We define six fine-grained metrics (Memory Utilization, Compute Utilization, Latency, Throughput, Energy Consumption, Compression Rate) to capture hardware saturation, latency-throughput balance, and carbon cost. Evaluating over 100 model-technique pairs (0.5B-72B parameters), we derive three core insights: (i) Efficiency involves quantifiable trade-offs: no single method is universally optimal; e.g., MoE reduces FLOPs and improves accuracy but increases VRAM by 40%, while int4 quantization cuts memory/energy by up to 3.9x at a 3-5% accuracy drop. (ii) Optima are task- and scale-dependent: MQA offers optimal memory-latency trade-offs for constrained devices, MLA achieves lowest perplexity for quality-critical tasks, and RSLoRA surpasses LoRA efficiency only beyond 14B parameters. (iii) Techniques generalize across modalities: we extend evaluations to Large Vision Models (Stable Diffusion 3.5, Wan 2.1) and Vision-Language Models (Qwen2.5-VL), confirming effective transferability. By open-sourcing datasets, evaluation pipelines, and leaderboards, EfficientLLM provides essential guidance for researchers and engineers navigating the efficiency-performance landscape of next-generation foundation models.

Analysis

Why This Paper Matters

As large language models grow in parameter count and context window, their computational, energy, and monetary costs become prohibitive. The AI community urgently needs systematic guidance on which efficiency techniques work best under which conditions. EfficientLLM fills this gap by providing the first comprehensive empirical study that evaluates a wide range of techniques—from efficient attention variants and sparse MoE to parameter-efficient fine-tuning and quantization—on a production-class cluster. This paper matters because it moves beyond isolated comparisons and reveals the nuanced trade-offs that practitioners must navigate: no single method is universally optimal, and the best choice depends on task, scale, and hardware constraints.

The study's scale (over 100 model-technique pairs, 0.5B-72B parameters) and its extension to vision and vision-language models make it a landmark reference. By open-sourcing datasets, evaluation pipelines, and leaderboards, EfficientLLM enables reproducible research and informed decision-making, directly addressing a critical need in the field.

Technical Contributions

  • Comprehensive benchmark: Covers three key axes—architecture pretraining (MQA, GQA, MLA, NSA, sparse MoE), fine-tuning (LoRA, RSLoRA, DoRA), and inference (int4, float16 quantization).
  • Six fine-grained metrics: Memory Utilization, Compute Utilization, Latency, Throughput, Energy Consumption, and Compression Rate, capturing hardware saturation, latency-throughput balance, and carbon cost.
  • Large-scale evaluation: Over 100 model-technique pairs on 48xGH200 and 8xH200 GPUs, spanning 0.5B to 72B parameters.
  • Cross-modality validation: Extends to Large Vision Models (Stable Diffusion 3.5, Wan 2.1) and Vision-Language Models (Qwen2.5-VL), confirming transferability.
  • Open-source resources: Datasets, evaluation pipelines, and leaderboards released to the community.

Results

Key quantitative findings include: MoE reduces FLOPs and improves accuracy but increases VRAM by 40%; int4 quantization cuts memory and energy by up to 3.9x with only a 3-5% accuracy drop. Task- and scale-dependent optima: MQA offers optimal memory-latency trade-offs for constrained devices, MLA achieves the lowest perplexity for quality-critical tasks, and RSLoRA surpasses LoRA efficiency only beyond 14B parameters. These results provide actionable guidance for practitioners.

Significance

EfficientLLM sets a new standard for evaluating efficiency techniques in foundation models. Its systematic approach and open-source resources will accelerate research and deployment of cost-effective AI systems. The study's insights into trade-offs and scale dependence are crucial for designing next-generation models that balance performance with compute, energy, and monetary costs, ultimately making AI more accessible and sustainable.

large-language-modelsefficiencybenchmarkattentionmixture-of-expertsquantizationfine-tuning
Large Language ModelsAttention MechanismsFine TuningDiffusion ModelsBenchmarksQuantization

Related Papers