Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
FreeAbout Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
This paper presents a two-player paradigm for enhancing LLM reasoning by separating reasoning and critique models. The critique model provides step-level feedback to supervise the reasoning model during both test-time and train-time. The authors introduce AutoMathCritique, an automated framework for collecting critique data, resulting in a dataset of 76,321 responses paired with step-level feedback. Fine-tuning language models with this dataset enables them to generate natural language feedback for mathematical reasoning. At test-time, critique models consistently improve the actor's performance on difficult queries, especially when scaling inference-time computation. The paper also proposes a critique-in-the-loop self-improvement method that enhances exploration efficiency and solution diversity on challenging queries, leading to stronger reasoning models. Code and datasets are publicly available.
Key Features
Pros & Cons
- Consistently improves reasoning performance on difficult queries at test-time
- Enhances exploration efficiency and solution diversity during self-training
- Automated and scalable data collection for critique data reduces human annotation effort
- Open-source code and datasets available for reproduction and further research
- Requires training a separate critique model in addition to the reasoning model
- Evaluation focused primarily on mathematical reasoning; generalization to other domains not extensively demonstrated
- Effectiveness may depend on the quality and coverage of the AutoMathCritique dataset
- Test-time scaling of inference computation may be computationally expensive