Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
FreeMeta-RL fine-tuning to optimize LLM test-time compute.
FreeFree tier
About Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
This paper introduces Meta Reinforcement Fine-Tuning (MRT), a novel method for optimizing test-time compute in large language models (LLMs) to improve reasoning performance. It formalizes test-time compute optimization as a meta-reinforcement learning problem, using cumulative regret over output tokens as a metric to balance exploration and exploitation. The method incorporates a dense reward bonus based on progress toward eventual success, leading to 2-3x relative performance gains and 1.5x token efficiency improvements on math reasoning tasks compared to standard outcome-reward RL.
Key Features
Formalizes test-time compute optimization as a meta-reinforcement learning problem
Uses cumulative regret over output tokens to measure test-time compute efficacy
Introduces a dense reward bonus based on progress (change in likelihood of eventual success)
Proposes Meta Reinforcement Fine-Tuning (MRT) method
Demonstrates 2-3x relative performance gain and 1.5x token efficiency improvement on math reasoning
Pros & Cons
Pros
- 2-3x relative performance gain over outcome-reward RL
- Roughly 1.5x gain in token efficiency
- Principled perspective on balancing exploration and exploitation at test time
- Formal framework using cumulative regret
Cons
- Evaluated only on math reasoning tasks, generalizability to other domains not shown
- Requires training with dense reward bonuses, adding complexity
- Currently a research method, not a readily deployable tool
Best For
Improving reasoning performance of large language modelsOptimizing test-time compute for math reasoning tasksExploring token efficiency in LLM inference
FAQ
What is Meta Reinforcement Fine-Tuning (MRT)?
MRT is a fine-tuning method that optimizes test-time compute for LLMs by formalizing it as a meta-reinforcement learning problem. It uses a dense reward bonus based on progress toward success to balance exploration and exploitation.
How does MRT compare to outcome-reward RL?
MRT achieves 2-3x relative performance gain and roughly 1.5x token efficiency improvement over standard outcome-reward RL on math reasoning tasks.
What metric does MRT use to evaluate test-time compute?
MRT uses cumulative regret over output tokens as a measure to assess the efficacy of test-time compute.