Swin Transformer V2

Analysis

Why This Paper Matters

Swin Transformer V2 tackles three core obstacles that prevented vision transformers from reaching the scale of their NLP counterparts: training instability as models grow, performance degradation when changing input resolutions, and the hunger for labeled data. By introducing targeted architectural fixes and a self-supervised pretraining strategy, the authors successfully train a 3-billion-parameter model that sets new records across image classification, object detection, semantic segmentation, and video action recognition. This work is a practical blueprint for scaling vision transformers efficiently.

How It Works

The original Swin Transformer introduced hierarchy, locality, and translation invariance to the vanilla Transformer encoder, using pre-normalization and relative position bias. However, scaling up model capacity and window resolution revealed two issues: activation values in deeper layers grew uncontrollably, causing training instability, and transferring position biases across different window sizes led to degraded performance.

Swin Transformer V2 addresses these with three key innovations:

Post-normalization and scaled cosine attention. Instead of normalizing before each sub-layer (pre-norm), the output of each residual block is normalized before being added back to the main branch. This prevents amplitude accumulation in deeper layers. For the largest models, an extra layer normalization is inserted every six blocks. Additionally, the standard dot-product attention is replaced with a scaled cosine function:

where the attention logit between pixel i and j is computed as a cosine similarity scaled by a learnable scalar τ (clipped above 0.01), with relative position bias B_ij added. This prevents a few pixel pairs from dominating the attention map in large models.

Continuous relative position bias with log-spaced coordinates. Rather than directly optimizing a parameterized bias matrix, a small meta-network G takes relative coordinates as input and outputs bias values. This allows the bias to adapt to different window sizes and tasks. To further improve transfer across resolutions, log-spaced coordinates are used instead of linear spacing, reducing the extrapolation ratio when moving to larger windows.

Self-supervised pretraining (SimMIM). To reduce dependence on labeled data, the model is pretrained using a masked image modeling approach. This enables the 3B model to be trained with only 70 million labeled images (1/40th of the JFT-3B dataset used by prior work).

Results

SwinV2-G achieves 86.8% top-1 accuracy on ImageNet V2, surpassing the previous best by 0.7%. On COCO object detection and instance segmentation, it outperforms prior results by +1.8 and +1.4 points respectively.

For semantic segmentation on ADE20K, SwinV2-G gains +1.5 points over the previous best, with an additional +0.2 from using larger windows at test time.

On Kinetics-400 video action classification, the model achieves 86.8% top-1 accuracy, a +1.4% improvement.

Significance

Swin Transformer V2 provides a clear recipe for scaling vision transformers to billions of parameters without sacrificing stability or requiring massive labeled datasets. The architectural modifications—post-norm, cosine attention, and log-spaced continuous position bias—are simple yet effective, and the self-supervised pretraining strategy makes large-scale vision models more accessible. This work has already influenced subsequent vision transformer designs and demonstrates that scaling laws observed in NLP can be replicated in computer vision with appropriate inductive biases.