FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

Analysis

Why This Paper Matters

World models have become a cornerstone for predicting physical dynamics, yet most existing evaluations focus on low-density, lane-structured settings. This paper identifies a critical gap: crowded, chaotic Global South urban environments, termed DENSEWORLD, which exhibit soft spatial boundaries, extreme agent heterogeneity, persistent occlusion, and rapid social negotiation. These conditions are far more representative of real-world urban complexity in many parts of the world, yet they remain largely unexplored. By introducing the first large-scale dataset for this regime, the authors enable research on a previously neglected but practically important scenario.

The paper also challenges the monolithic latent approach of existing JEPA formulations. In dense scenes, a single latent vector struggles to preserve fine-grained interaction dynamics and handle partial observability. FactorJEPA's factorization into layout, agent, and interaction channels is a principled response, making world structure a first-class predictive primitive. This shift could influence how future world models are designed, moving beyond monolithic encodings to structured, interpretable representations.

Technical Contributions

• DENSEWORLD-115k dataset: 1,000 hours of drive-through, walk-through, and aerial video across 22 cities, providing a rich benchmark for crowded urban scenes.
• FactorJEPA architecture: Decomposes the future latent into three separated subspaces—layout, entities, and interactions—each capturing distinct aspects of the scene.
• Visibility gate: A mechanism to preserve partially observed agents, addressing the challenge of occlusion that is pervasive in dense environments.
• Cross-factor shortcut prevention: Separated subspaces discourage the model from taking shortcuts between factors, promoting more robust and causal predictions.
• Comprehensive evaluation: Metrics include Future-frame L1, Causal L1, Mask-ratio slope, and Motion cosine, offering a multi-faceted assessment of prediction quality.

Results

The paper reports that FactorJEPA improves future-latent accuracy (Future-frame L1), intervention-sensitive prediction (Causal L1), and robustness to reduced visual evidence (Mask-ratio slope) compared to existing JEPA formulations. It also exposes a reproducible motion-information trade-off (Motion cosine), indicating that the model can balance between preserving motion details and compressing information. Notably, the method rankings replicate across 2B and 1B V-JEPA 2.1 backbones, with Spearman rho values between 0.895 and 0.978, demonstrating consistency and scalability. These results suggest that FactorJEPA is not only effective but also stable across model sizes.

Significance

This work has broad implications for AI systems operating in real-world urban environments, particularly in the Global South where dense, chaotic traffic is the norm. By providing a dataset and a model that handles such complexity, the paper could accelerate progress in autonomous driving, urban planning, and robotics. The factorization approach also offers a new direction for world model research, emphasizing structured representations over monolithic latents. The public release of the dataset and checkpoints will facilitate further research and reproducibility, making this a valuable contribution to the community.