Parameter-efficient fine-tuning in large language models: a survey of methodologies

Analysis

Why This Paper Matters

As large language models (LLMs) grow to hundreds of billions of parameters, full fine-tuning becomes prohibitively expensive in terms of compute and memory. This survey addresses a critical bottleneck in deploying LLMs to diverse downstream tasks by systematically reviewing parameter-efficient fine-tuning (PEFT) methods. It provides a much-needed structured overview for practitioners who need to adapt LLMs without incurring the cost of full retraining.

The paper is particularly timely given the rapid adoption of LLMs in industry and academia. By categorizing methods into additive (e.g., adapters), selective (e.g., bitfit), and reparameterization-based (e.g., LoRA) approaches, it offers a clear framework for understanding the trade-offs between parameter savings and task performance. This helps practitioners make informed decisions when choosing a PEFT strategy.

Technical Contributions

The survey's main technical contribution is its comprehensive taxonomy of PEFT methods, which includes:

• Additive methods: Inserting small trainable modules (e.g., adapter layers, prefix tuning) while keeping the base model frozen.
• Selective methods: Fine-tuning only a subset of existing parameters (e.g., bias terms, specific layers).
• Reparameterization methods: Using low-rank approximations (e.g., LoRA) to reduce the number of trainable parameters.

The paper also discusses hybrid approaches and compares methods based on parameter efficiency, memory footprint, and inference overhead. It highlights that LoRA and adapter-based methods are among the most popular due to their balance of efficiency and performance.

Results

While the survey does not present new experimental results, it synthesizes findings from prior work. Key reported observations include:

• PEFT methods typically require only 0.1% to 10% of the parameters of full fine-tuning.
• Performance on many NLP benchmarks (e.g., GLUE, SuperGLUE) is within 1-2% of full fine-tuning.
• LoRA and adapter methods often outperform selective methods on complex tasks, while selective methods like BitFit are extremely parameter-efficient but may underperform on harder tasks.

The survey notes that the choice of PEFT method depends on factors such as model size, task complexity, and available compute.

Significance

This survey has significant practical value for the AI community. It democratizes access to LLM adaptation by lowering the computational barrier, enabling smaller organizations and researchers to fine-tune large models. It also identifies open research directions, such as developing automated methods for selecting the best PEFT strategy and understanding the theoretical underpinnings of why PEFT works. As LLMs continue to scale, PEFT will become increasingly essential, making this survey a foundational reference for future work.