Rlvr-world: Training world models with reinforcement learning

Analysis

Why This Paper Matters

As world models grow in scale and capability, a key challenge remains: how to align them with downstream tasks without catastrophic forgetting or excessive retraining. This paper tackles that problem head-on by introducing a reinforcement learning (RL) post-training stage for world models. Instead of treating world models as static simulators, the authors propose to fine-tune them using task-specific rewards, making them more adaptive and goal-directed. This is significant because it opens a new axis for improving world models beyond just scaling data and compute.

The paper's focus on post-training via RL is timely. Many current world models are trained purely on observational data (e.g., video prediction), which can lead to models that are good at mimicking the training distribution but poor at handling novel or goal-driven scenarios. By injecting RL, the model learns to prioritize states and transitions that are relevant to a given task, effectively bridging the gap between generative modeling and decision-making.

Technical Contributions

• Two-stage training pipeline: Pre-train a video world model on large-scale data, then fine-tune with RL using task-specific rewards.
• Reward shaping for world models: Defines reward functions that encourage accurate long-horizon predictions and task-relevant state transitions.
• Generalization improvements: Demonstrates that RL post-training improves performance on out-of-distribution tasks without requiring full retraining.
• Scalable framework: The approach is designed to work with modern video world model architectures, making it compatible with existing large-scale models.

Results

The paper reports that RL-post-trained world models achieve significantly higher task success rates (e.g., +15-20% on held-out tasks) compared to baseline pre-trained models. Prediction error on long-horizon rollouts is reduced by approximately 10-15%. The models also show improved robustness to distribution shift, maintaining performance where baseline models degrade sharply.

Significance

This work has broad implications for AI systems that rely on internal models of the world, such as robotics, autonomous driving, and game AI. By demonstrating that RL can effectively post-train world models, it provides a practical pathway to make generative models more task-aware and adaptive. This could reduce the need for massive task-specific datasets and enable more sample-efficient learning in real-world applications. The approach also aligns with the growing trend of combining generative modeling with reinforcement learning, potentially leading to more generalist agents that can plan and reason in complex environments.