Critique Fine-Tuning

Analysis

Why This Paper Matters

Supervised fine-tuning (SFT) has been the go-to method for adapting language models to specific tasks, but it has a fundamental flaw: it forces models to imitate correct answers, often without fostering deeper understanding. Critique Fine-Tuning (CFT) challenges this paradigm by training models to critique noisy responses, much like how humans learn through critical analysis. This shift is significant because it leverages abundant, imperfect data—such as web-sourced educational content—rather than requiring meticulously curated correct examples. The paper shows that learning to critique not only improves reasoning but also leads to faster convergence and better data efficiency, making it a practical advance for real-world applications where high-quality data is scarce.

How It Works

CFT is inspired by human learning processes that emphasize critical thinking. Instead of training a model to generate a correct answer from a question, CFT provides the model with both the question and a noisy response, then asks it to produce a critique of that response. The training objective is straightforward: given input x (question) and y (noisy response), the model parameters are optimized to generate critique c using a standard language modeling loss.

The dataset, WebInstruct, is a large collection of instruction-response pairs from online educational resources and quiz websites, spanning mathematics (65%), physics, chemistry, business, and humanities. The responses are extracted and refined by large language models like Qwen-72B and Mixtral, making them highly prone to noise. From this, the authors create several subsets:

• WebInstruct-SFT: 50K samples directly from the original dataset, with over 50% error rate.
• WebInstruct-verified: 50K samples where GPT-4o judged the original answers as correct.
• WebInstruct-GPT-4o: 50K samples with questions from SFT but answers generated by GPT-4o.
• WebInstruct-CFT: 50K samples where GPT-4o provides detailed critiques of the original responses. About 56% are judged correct, the rest wrong.
• WebInstruct-CFT-Tiny: A 4K subset for training 32B models.

The training loss is:

The authors compare three SFT settings (on noisy responses, verified responses, and GPT-4o-generated responses) against CFT, using MATH-500 as a validation set.

Results

CFT consistently outperforms all SFT baselines across different base models on mathematical reasoning benchmarks. For example, on DeepSeek-Math-7B, CFT achieves a 3.5% absolute improvement over SFT-GPT4o. On Qwen2.5-7B, CFT shows a substantial 10.4% improvement over SFT-verified. On Qwen2.5-Math-7B, CFT surpasses SFT-GPT4o by 6.7%. Qwen2.5-Math-7B with CFT achieves the best performance among 7B models.

CFT demonstrates faster convergence and maintains higher performance throughout training compared to SFT variants. It achieves approximately 80% accuracy on MATH and 40% on Minerva-Math, while SFT variants lag behind.

The 7B CFT model (Qwen2.5-Math-7B-CFT) achieves the highest average performance (48.1%) among 7B-scale models with significantly less training data (50K samples). It outperforms specialized math models like Deepseek-Math-7B-Instruct, Mathstral-7B, and NuminaMath-7B-CoT. Remarkably, it shows competitive performance compared to much larger models like Qwen2.5-Math-72B-Instruct.

The 32B CFT model (Qwen2.5-32B-Instruct-CFT) is highly data-efficient, achieving optimal performance with only 4K training samples, compared to 17K for Sky-T1-32B-Preview.

CFT is effective even with a modest critique model (GPT-4o-mini), and performance improves further with a stronger critique model (GPT-4o-1120).

Significance

Critique Fine-Tuning represents a paradigm shift in how we train language models for reasoning tasks. By focusing on critique generation rather than answer imitation, CFT leverages noisy, abundant data more effectively, reducing the need for expensive human annotation. The method's data efficiency—especially the 32B model achieving top results with only 4K samples—suggests that smaller models can compete with much larger ones when trained with the right objective. This could democratize access to high-performing reasoning models, as organizations with limited compute can achieve strong results with less data. The paper also opens avenues for further research: can CFT be combined with reinforcement learning or iterative self-critique? Does it generalize to other domains like code generation or scientific reasoning? The findings challenge the dominance of SFT and invite the community to rethink how we define 'learning' in language models.