SAM 2

Analysis

Why This Paper Matters

Segment Anything Model 2 (SAM 2) represents a significant leap in visual segmentation by extending the promptable paradigm from static images to dynamic videos. While the original SAM revolutionized image segmentation with its ability to segment any object given a prompt, SAM 2 tackles the far more challenging domain of video, where objects move, occlude, and change appearance over time. This is crucial for real-world applications like video editing, autonomous driving, and augmented reality, where understanding objects across frames is essential. By introducing a streaming memory mechanism, SAM 2 achieves near real-time performance, making it practical for interactive use.

How It Works

SAM 2 is built on a simple transformer architecture that generalizes SAM to video. The core innovation is the memory attention mechanism, which conditions the current frame's features on past predictions and prompts stored in a memory bank. This allows the model to propagate segmentation masks across frames while incorporating user corrections.

The architecture consists of:

• Image Encoder: A pre-trained MAE Hiera model that produces hierarchical features. The stride 16 and 32 features are fused into a pyramid for memory attention, while stride 4 and 8 features are used via skip connections for high-resolution mask decoding.
• Memory Attention: A stack of 4 transformer blocks, each performing self-attention on the current frame embedding, cross-attention to memories (spatial features and object pointers from the memory bank), and an MLP. It uses sinusoidal absolute positional embeddings for temporal information and 2D spatial RoPE for spatial attention.
• Prompt Encoder and Mask Decoder: Identical to SAM's, supporting clicks, boxes, and masks. The decoder uses two-way transformer blocks to update prompt and frame embeddings. For ambiguous prompts, it predicts multiple masks and selects the one with highest predicted IoU. An occlusion prediction head outputs a visibility score to handle frames where the object is absent.
• Memory Encoder and Memory Bank: The memory encoder downsamples the predicted mask and sums it with the unconditioned frame embedding, then applies lightweight convolutions. The memory bank maintains FIFO queues for recent frames (up to N) and prompted frames (up to M), plus object pointer tokens. Temporal positional embeddings are added only to recent frames to capture motion.

Results

SAM 2 was evaluated on 9 zero-shot video datasets in both offline and online interactive settings. It consistently outperformed baselines (SAM+XMem++ and SAM+Cutie) in J&F accuracy, achieving higher quality with 3× fewer user interactions. In image segmentation, it is 6× faster and more accurate than SAM. The model also excels in semi-supervised VOS with click, box, or mask prompts on the first frame.

Significance

SAM 2's impact extends beyond its immediate performance gains. By unifying image and video segmentation under a single promptable framework, it simplifies the development of interactive tools for video editing and analysis. The release of the SA-V dataset (50.9K videos, 642.6K masklets) provides a valuable resource for future research. The model's real-time capability and flexibility in handling various prompt types make it a strong foundation for downstream applications. However, its reliance on clearly defined object boundaries and the need for careful occlusion handling are areas for future improvement. Overall, SAM 2 pushes the boundary of what is possible with promptable visual segmentation, bringing us closer to truly interactive video understanding.