Speculative decoding and beyond: An in-depth survey of techniques

Analysis

Why This Paper Matters

As large language models (LLMs) grow in size, inference latency becomes a critical bottleneck for real-time applications. Speculative decoding has emerged as a powerful paradigm to accelerate autoregressive generation by using a small draft model to propose multiple tokens, which are then verified in parallel by the large target model. This survey is timely, as it consolidates the rapidly expanding body of work in this area, providing a clear taxonomy that helps researchers navigate the landscape.

The paper goes beyond traditional speculative decoding by also covering iterative refinement methods inspired by numerical optimization. This broader perspective is valuable because it highlights the underlying mathematical principles that connect seemingly disparate acceleration techniques. For AI practitioners, understanding these connections can inspire novel approaches to speed up inference in their own systems.

Technical Contributions

The survey's main contribution is its comprehensive categorization of acceleration techniques. Key innovations covered include:

• Draft model-based speculative decoding: Methods that use a smaller, faster model to generate candidate tokens, which are then checked by the larger model. This includes variations like single-draft and multi-draft strategies.
• Iterative refinement: Techniques that iteratively improve the output sequence, often drawing parallels to numerical optimization methods like gradient descent. These methods can refine the entire sequence in parallel, reducing the number of sequential steps.
• Early exiting and layer skipping: Approaches that allow the model to exit early for easier tokens, saving computation without sacrificing quality.
• Parallel verification: The core mechanism that enables speculative decoding to achieve speedups by validating multiple tokens at once.

The survey also discusses the common principles that make these methods effective, such as the trade-off between draft model quality and speed, and the importance of verification strategies.

Results

Since this is a survey, it does not present new experimental results. However, it synthesizes findings from the literature, noting that speculative decoding can achieve 2-3x speedups in many cases, with some methods reporting even higher gains. The survey emphasizes that these speedups are achieved while maintaining the same output distribution as the target model, which is a critical property for deployment. It also notes that iterative refinement methods can offer additional gains in certain settings, though they may require more careful tuning.

Significance

The broader impact of this survey lies in its potential to accelerate the adoption of inference acceleration techniques in real-world AI systems. By providing a clear overview, it lowers the barrier to entry for practitioners who may be unfamiliar with these methods. Furthermore, by highlighting open challenges, it sets the stage for future research. As LLMs continue to scale, efficient inference will be a key differentiator, and this survey serves as a foundational resource for anyone working on or with large generative models.