Reasoning on a budget: A survey of adaptive and controllable test-time compute in llms

Analysis

Why This Paper Matters

As large language models (LLMs) are deployed at scale, inference cost has become a critical bottleneck. Test-time compute (TTC) — the additional computation performed during inference to improve output quality — offers a way to trade compute for accuracy. However, naive TTC strategies can be prohibitively expensive. This survey addresses the need for adaptive and controllable TTC, where the amount of compute is dynamically adjusted based on the difficulty of the input or the desired performance level.

The paper is timely because recent models like OpenAI's o1 have popularized 'thinking' tokens, but the computational overhead is substantial. By surveying methods that make TTC more efficient, this work provides a roadmap for deploying high-quality LLMs within budget constraints. It also unifies terminology and paradigms, which is valuable for a rapidly evolving field.

Technical Contributions

The survey categorizes TTC methods into several paradigms, including parallel and sequential compute, and analyzes how each can be made adaptive. Key innovations highlighted include:

• Adaptive compute allocation: Techniques that decide how much compute to use per token or per query, often based on confidence scores or difficulty estimators.
• Controllable compute: Methods that allow users to specify a compute budget or a target performance level, enabling a trade-off between latency and accuracy.
• Efficient parallel TTC: Strategies that generate multiple candidate responses in parallel and then select or aggregate them, reducing wall-clock time compared to sequential refinement.
• Early-exit and dynamic depth: Approaches that adjust the number of transformer layers used during inference, saving compute for easier inputs.
• Reinforcement learning and meta-learning: Training models to decide when to stop generating or when to invoke additional reasoning steps.

Results

As a survey, the paper does not present new experimental results. Instead, it synthesizes findings from the literature, noting that adaptive TTC can reduce inference cost by a significant margin (e.g., up to 50% in some cited works) while maintaining comparable accuracy. The survey also highlights that the efficiency gains vary across paradigms: parallel TTC often provides better latency but higher total compute, while sequential TTC can be more compute-efficient but slower. The authors emphasize that the choice of paradigm depends on the application's latency and cost constraints.

Significance

This survey is a valuable resource for AI practitioners and researchers aiming to deploy LLMs cost-effectively. By organizing the landscape of adaptive TTC, it lowers the barrier to entry for those new to the field and helps experts identify gaps. The emphasis on controllability aligns with the growing demand for user-specified trade-offs between quality and cost. As LLMs become ubiquitous, methods that enable 'reasoning on a budget' will be essential for sustainable AI deployment. This work likely spurs further research into unified frameworks for adaptive compute and may influence the design of future inference systems.