SceneActBench: Can Agents Act on the 3D Scenes They See?

Analysis

Why This Paper Matters

SceneActBench addresses a critical gap in evaluating vision-language models (VLMs) for embodied AI: the ability to act on complete 3D scenes rather than just describe them or manipulate single objects. Existing benchmarks like ScanQA or EmbodiedQA focus on textual responses or simple object interactions, leaving multi-object, task-oriented action under-evaluated. As VLMs increasingly integrate with robotic and simulation tools, a standardized benchmark for visually conditioned 3D action becomes essential. This paper provides that benchmark, revealing that even top proprietary models struggle to perform consistently across diverse 3D tasks.

The significance lies in its unified evaluation framework. By fixing the agent loop and using task-specific geometric metrics, SceneActBench enables fair comparison across models and tasks. The finding that overall scores range only from 38.6 to 50.2 underscores the difficulty of this problem and the need for further research. This benchmark will likely become a reference point for future VLM agent development in 3D environments.

Technical Contributions

• Unified agent-environment loop: All tasks run through one fixed loop, ensuring fair comparison across models and tasks.
• Five diverse 3D tasks: Built from 210 source instances yielding 520 task cases, covering a range of action types (e.g., navigation, manipulation, rearrangement).
• Task-specific geometric metrics: Each task has a hidden ground truth and tailored metric (e.g., success rate, distance error, overlap), avoiding reliance on generic scores.
• Input flexibility: Agents can use PNG images, sampled video frames, or supplied 3D assets, reflecting real-world deployment scenarios.
• Failure analysis: The paper dissects where and how failures occur, providing actionable insights for model improvement.

Results

Eleven proprietary VLM configurations were evaluated. Overall scores span 38.6 to 50.2, with no model achieving consistent high performance across all five tasks. This indicates that current VLMs lack robust 3D scene understanding and action planning. The failure analysis likely reveals common issues such as misinterpreting spatial relationships, failing to generalize to novel object arrangements, or poor integration of visual and action modalities. Concrete per-task scores are not provided in the abstract, but the overall range highlights significant room for improvement.

Significance

SceneActBench sets a new standard for evaluating VLM agents in 3D environments. It moves beyond static description or single-object tasks to realistic multi-object scenarios, aligning with the goals of embodied AI and robotics. The benchmark's design—fixed loop, geometric metrics, diverse tasks—makes it reproducible and extensible. For the AI community, it identifies a clear performance gap and provides a testbed for developing more capable agents. Future work can build on this benchmark to improve spatial reasoning, action planning, and cross-task generalization in VLMs.