Neura Market
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking logo

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

Free

Small LLMs master math reasoning with self-evolved deep thinking

FreeFree tier
Inputs: textOutputs: text
Type
Open Source
Visit Website GitHub (40 stars)

About rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

rStar-Math is a method for small language models (SLMs) that enables them to achieve state-of-the-art math reasoning capabilities, rivaling or surpassing larger models like OpenAI o1, without relying on distillation from superior models. The approach leverages Monte Carlo Tree Search (MCTS) to perform 'deep thinking' during test time, guided by a process preference model (PPM) trained from scratch. It introduces three key innovations: a code-augmented chain-of-thought data synthesis method that generates verified reasoning trajectories, a novel process reward training approach that avoids naive step-level annotations, and a self-evolution recipe where the policy SLM and PPM iteratively improve over four rounds. Using millions of synthesized solutions for 747k math problems, rStar-Math boosts performance on the MATH benchmark from 58.8% to 90.0% on Qwen2.5-Math-7B and from 41.4% to 86.4% on Phi3-mini-3.8B, also solving 53.3% of AIME problems. The method is designed to be open-source, with code and data planned for release.

Key Features

Monte Carlo Tree Search (MCTS) for deep thinking during test time
Code-augmented chain-of-thought data synthesis for verified reasoning trajectories
Process Preference Model (PPM) trained without naive step-level annotations
Self-evolution recipe that iteratively improves policy SLM and PPM
Achieves state-of-the-art math reasoning on small models (e.g., 3.8B Phi3-mini)
Outperforms OpenAI o1-preview on MATH and AIME benchmarks
No distillation from larger models required

Pros & Cons

Pros
  • Enables small models to surpass much larger models like o1-preview
  • No need for distillation from larger, proprietary models
  • Open-source methodology with planned code and data release
  • Iterative self-evolution leads to significant performance gains
  • Transparent three-innovation approach detailed in the paper
Cons
  • May require substantial computational resources for MCTS and self-evolution
  • Currently a research method, not a plug-and-play tool
  • Limited to math reasoning domain; generalizability not shown
  • Code and data were not yet publicly available at the time of paper submission

Best For

Improving math reasoning capabilities of small language modelsSolving competition-level math problems (e.g., USA Math Olympiad AIME)Enhancing performance on mathematical benchmarks like MATHResearch on self-evolution and test-time search for reasoning

FAQ

What is rStar-Math?
rStar-Math is a method that enables small language models (SLMs) to achieve advanced math reasoning through Monte Carlo Tree Search and a self-evolution process, without distillation from larger models.
How does rStar-Math work?
It uses a math policy SLM to perform test-time search guided by a process preference model (PPM) trained via a novel code-augmented data synthesis method. Both models are iteratively evolved over four rounds using millions of synthesized math solutions.
What benchmarks does rStar-Math perform well on?
On the MATH benchmark, it improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to 86.4%, outperforming o1-preview. On the USA Math Olympiad (AIME), it solves an average of 53.3% of problems.