Autoregressive Image Models (AIM)

Analysis

Why This Paper Matters

This paper is significant because it demonstrates that autoregressive pretraining, the dominant paradigm in large language models (LLMs), can be effectively applied to visual data at scale. While masked image modeling (e.g., MAE) and contrastive learning (e.g., CLIP) have been the go-to methods for vision, AIM shows that a simple next-patch prediction objective, combined with appropriate architectural tweaks, scales with model size and data volume in a manner analogous to LLMs. This is important because it suggests that the same training recipe used for language can work for vision, potentially simplifying the development of multimodal models that share a common autoregressive backbone.

The paper also provides evidence that autoregressive vision models can match or exceed the performance of more complex self-supervised methods, especially at larger scales. This challenges the assumption that bidirectional context is necessary for strong visual representations and opens the door to using highly optimized LLM training infrastructure for vision tasks.

Technical Contributions

• Autoregressive objective for images: The model predicts the next patch in a raster-scan order, treating an image as a sequence of patches. This is a direct analog of next-token prediction in language models.
• Prefix attention: To allow the model to see the full context of already predicted patches (not just causal), the authors use a prefix attention mask where the first few patches attend bidirectionally, and the rest attend causally. This helps stabilize training and improves representation quality.
• Parameterized prediction head: Instead of a simple linear projection, the prediction head is a small MLP that maps the hidden state to patch-level predictions. This increases capacity and improves performance.
• Scaling study: The authors train models ranging from 300M to 8B parameters on up to 2B images, showing consistent improvements in downstream task accuracy with model size and data volume.

Results

The paper reports that the 8B AIM model achieves strong performance on ImageNet-1K classification (top-1 accuracy) and other vision tasks, though exact numbers are not provided in the abstract. The key result is the demonstration of scaling laws: performance improves log-linearly with model size and data size, similar to what is observed in LLMs. The authors also show that the autoregressive objective outperforms a masked image modeling baseline at the same scale, and that prefix attention is crucial for achieving good results.

Significance

AIM has the potential to unify vision and language pretraining under a single autoregressive framework. This could simplify the architecture and training of multimodal models, reduce engineering overhead, and allow vision to benefit from the rapid advances in LLM infrastructure (e.g., efficient attention, distributed training). The paper also reinforces the idea that scaling alone can drive progress in vision, reducing the need for task-specific architectural innovations. However, the lack of direct comparisons to state-of-the-art methods like DINOv2 or CLIP on standard benchmarks limits the immediate practical impact. Future work will need to show whether autoregressive vision can match or exceed these methods in sample efficiency and final accuracy.