From word to world: Can large language models be implicit text-based world models?

Analysis

Why This Paper Matters

This paper tackles a fundamental question: can large language models (LLMs) go beyond pattern matching to actually model the world? By proposing a systematic evaluation of LLMs as implicit text-based world models, it addresses a critical gap in understanding whether these models possess genuine reasoning capabilities or merely mimic surface-level statistics. The findings have implications for AI safety, planning, and interactive agents that must understand environment dynamics.

The work is timely given the rapid deployment of LLMs in decision-making roles. If LLMs can reliably model world dynamics from text, they could serve as lightweight simulators for training other AI systems or for few-shot planning in novel environments. This paper provides the first structured attempt to measure that capability.

Technical Contributions

The paper introduces two key evaluation metrics:

• Fidelity: Measures how accurately the LLM predicts state transitions in a text-based environment, compared to ground-truth dynamics.
• Coherence: Assesses whether the model's predictions remain internally consistent over multiple steps, avoiding contradictions.
• The authors design a benchmark of text-based environments (e.g., grid worlds, simple physics) and test multiple LLM sizes and training regimes.
• They show that fidelity and coherence improve with model scale and training data quality, suggesting these are learnable properties.

Results

• Larger models (e.g., 70B parameters) achieve >90% fidelity on simple environments, while smaller models (7B) plateau around 70%.
• Coherence scores correlate strongly with fidelity (r=0.85), indicating that accurate models also maintain consistency.
• Models trained on diverse text corpora outperform those trained on narrow domains, highlighting the importance of broad world knowledge.
• The best-performing LLM matches a specialized world model baseline on 80% of tasks, suggesting LLMs can approximate dedicated simulators.

Significance

This research opens the door to using LLMs as general-purpose world models for text-based reasoning, planning, and simulation. It could reduce the need for hand-crafted environment models in AI training pipelines. However, the limitation to text-based settings means real-world applications require further validation. The proposed metrics provide a foundation for future work on grounding LLMs in physical or multimodal environments.