BroRL: Scaling Reinforcement Learning via Broadened Exploration
FreeScaling reinforcement learning via broadened exploration for LLMs
FreeFree tier
Inputs: textOutputs: text
About BroRL: Scaling Reinforcement Learning via Broadened Exploration
BroRL (Broadened Reinforcement Learning) is a method for scaling reinforcement learning in large language models by increasing the number of rollouts per training example to hundreds, enabling broader exploration. It complements prior work that scaled training steps (ProRL) and overcomes performance plateaus. BroRL is motivated by a mass balance equation analysis showing that increasing rollouts ensures correct token probability mass expansion. Empirically, it revives models that saturated after 3K ProRL training steps and achieves state-of-the-art results for 1.5B parameter models across diverse benchmarks.
Key Features
Increases rollouts per example to hundreds for broader exploration
Mass balance equation analysis to ensure correct token probability mass expansion
Revives models saturated after 3K ProRL training steps
Achieves state-of-the-art results for 1.5B models across diverse benchmarks
Complementary to scaling training steps in ProRL
Pros & Cons
Pros
- Continuous performance improvement beyond saturation point of step-scaling
- Theoretical foundation via mass balance equation analysis
- Demonstrated SOTA on 1.5B parameter models
- Compatible with existing RLVR frameworks
Cons
- Requires substantial computational resources for hundreds of rollouts per example
- Empirical results only shown for 1.5B models; scalability to larger models not yet demonstrated
- May introduce latency due to increased rollout count
Best For
Improving reasoning capabilities in large language modelsOvercoming performance plateaus in reinforcement learning trainingScaling RL beyond increasing training stepsResearch on exploration strategies in RLVR
FAQ
What is BroRL?
BroRL is a reinforcement learning method that scales RL for LLMs by broadening exploration via many rollouts per training example, as opposed to scaling the number of training steps.
How does BroRL differ from ProRL?
ProRL scales RL by increasing training steps, which plateaus after thousands of steps. BroRL instead increases the number of rollouts per example to hundreds, yielding continued gains.
What is the theoretical basis for BroRL?
BroRL is motivated by a mass balance equation analysis showing that increasing the number of rollouts per example diminishes the effect of unsampled tokens, ensuring expansion of correct token probability mass.
What results has BroRL achieved?
BroRL revives models that saturated after 3K ProRL training steps and achieves state-of-the-art results for the 1.5B model across diverse benchmarks.