An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models

Free

Compute-optimal inference scaling laws for LLMs

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models

This research paper empirically studies inference scaling laws (test-time scaling laws) for large language models (LLMs), focusing on compute-optimal inference configurations. The authors analyze trade-offs between model size and the amount of compute used for generating additional tokens via various inference strategies, including greedy search, majority voting, best-of-n, weighted voting, and two tree search algorithms. Key findings indicate that scaling inference compute with advanced strategies can be more computationally efficient than scaling model parameters. For instance, a smaller Llemma-7B model paired with a novel tree search algorithm consistently outperforms the larger Llemma-34B model across all tested strategies on the MATH benchmark. The work provides insights into Pareto-optimal trade-offs between cost and performance, aiming to guide the design of compute-optimal inference methods for problem-solving tasks.

Key Features

Study of inference scaling laws (test-time scaling laws) for LLMs

Analysis of compute-optimal inference trade-offs between model size and inference compute

Evaluation of multiple inference strategies: greedy search, majority voting, best-of-n, weighted voting, and tree search algorithms

Novel tree search algorithm demonstrating Pareto-optimal performance with smaller models

Empirical results on the MATH benchmark using Llemma models (7B and 34B)

Pros & Cons

Pros

Demonstrates that scaling inference compute can be more efficient than scaling model parameters
Smaller models with advanced inference can outperform larger models, reducing hardware costs
Provides a systematic framework for compute-optimal inference decision-making
Covers a range of common inference strategies for practical guidance

Cons

Empirical results are limited to the MATH benchmark and Llemma model family
Does not cover all possible inference strategies or diverse task domains
Findings may not directly generalize to other model architectures or tasks without further validation
Focus on problem-solving; applicability to generative or conversational tasks is not explored

Best For

Optimizing inference compute for large language model deploymentsProblem-solving tasks requiring cost-performance trade-offsMath reasoning and benchmark evaluation (e.g., MATH)Designing efficient inference strategies for resource-constrained environments

FAQ

What are inference scaling laws?

Inference scaling laws, also called test-time scaling laws, refer to the relationship between the amount of compute used during inference (e.g., generating additional tokens or using multiple samples) and the resulting performance of an LLM. This paper studies how to optimally allocate inference compute.

What inference strategies were studied?

The paper evaluates greedy search, majority voting, best-of-n, weighted voting, and two different tree search algorithms.

What is the main finding of the paper?

The main finding is that scaling inference compute with advanced inference strategies can be more computationally efficient than scaling model parameters. For example, a Llemma-7B model with a novel tree search algorithm outperforms the Llemma-34B model on the MATH benchmark.