LeViT

Analysis

Why This Paper Matters

Vision transformers (ViTs) have revolutionized image classification by applying attention mechanisms to image patches, but their computational cost often makes them impractical for real-time or edge deployment. LeViT addresses this gap by blending convolutional and transformer elements, achieving state-of-the-art speed-accuracy tradeoffs. This paper is significant because it shows that careful architectural choices—like using convolutional patch embeddings, attention bias instead of positional embeddings, and reduced MLP expansion—can make transformers competitive with efficient convnets like EfficientNet on CPU, while retaining the flexibility of attention. For practitioners, LeViT offers a practical path to deploying high-accuracy models in latency-sensitive applications.

How It Works

LeViT builds on the ViT architecture and DeiT training method but introduces several key modifications for efficiency. The input image is first processed by four convolutional layers (3x3, stride 2) that reduce resolution while increasing channel depth from 3 to 256, producing a feature map of shape (256, 14, 14) with only 184 MFLOPs. This patch embedding replaces the standard ViT patch splitting, leveraging convolution's efficiency.

Instead of a classification token, LeViT uses average pooling on the final activation map, similar to convnets, which allows the model to work with BCHW tensor format and reduces overhead. During training, separate heads are trained for distillation, and at test time their outputs are averaged. Normalization uses batch normalization after each convolution, and all activations are Hardswish, a computationally cheaper alternative to GELU.

The core innovation is the multi-resolution pyramid: LeViT integrates ResNet-like stages into the transformer. Between stages, a shrinking attention block downsamples the activation map by subsampling before the Q transformation. To provide positional information, an attention bias is added to the attention maps, explicitly encoding relative positions. This bias reduces the need for keys to carry location data, allowing the keys to be smaller (D ∈ {16, 32}) while the value matrix V has 2D channels. For downsampling layers without residual connections, V is set to 4D to prevent information loss. The MLP blocks are simplified to 1x1 convolutions with batch normalization and an expansion factor of 2 (instead of 4), further reducing computational cost.

Results

LeViT models are named by the input channels to the first transformer stage (e.g., LeViT-256). Each stage consists of attention and MLP block pairs, with shrinking attention blocks between stages. Training on ImageNet-2012 and evaluation on its validation set show clear advantages.

Key results: LeViT-384 matches DeiT-Small accuracy with half the FLOPs. LeViT-128S matches DeiT-Tiny with 4x fewer FLOPs. LeViT-192 and LeViT-256 achieve similar accuracy to EfficientNet B2 and B3 but are 5x and 7x faster on CPU, respectively. On alternative test sets like ImageNet-Real and ImageNet-V2, LeViT-256 matches EfficientNet B3 on Real but is slightly worse (-0.6) on V2, while LeViT-384 is slightly behind DeiT-Small on both. Despite this, the speed-accuracy tradeoff remains favorable.

Compared to other ViT variants, LeViT is significantly faster. Token-to-token ViTs require ~5x more FLOPs for similar accuracy. Bottleneck transformers and pyramid vision transformers are about 5x slower than LeViT-192 at comparable accuracy. Pooling-based ViT (PiT) and CvT come closest but are still 1.2x to 2.4x slower. LeViT's advantage stems from DeiT-style distillation, which boosts accuracy when training on ImageNet alone.

Significance

LeViT's practical impact is clear: it enables high-accuracy image classification on CPU with inference speeds that were previously only possible with lightweight convnets. By rethinking every component—from patch embedding to attention mechanism—the authors show that transformers can be made efficient without sacrificing accuracy. The hybrid approach, combining convolution's speed with attention's flexibility, points toward a future where vision transformers are viable for real-world deployment. For AI practitioners, LeViT offers a strong baseline for latency-critical applications, and its design principles can inspire further optimizations in other domains.