DeiT

Analysis

Why This Paper Matters

DeiT addresses a critical bottleneck in vision transformers: their data hunger. While ViT required massive datasets like JFT-300M to outperform CNNs, DeiT demonstrates that with a clever teacher-student distillation strategy, a pure transformer can achieve competitive results on ImageNet alone. This is significant because it lowers the barrier for adopting transformers in vision tasks, making them accessible to researchers and practitioners without access to proprietary datasets.

The paper also challenges the prevailing assumption that convolutions are necessary for strong visual representations. By achieving 83.1% top-1 accuracy on ImageNet with a convolution-free architecture, DeiT shows that attention mechanisms alone can capture spatial hierarchies effectively when guided by a strong teacher. This has implications beyond classification, potentially influencing object detection, segmentation, and video understanding.

Technical Contributions

• Distillation Token: A novel token added to the input sequence that interacts with both the class token and patch tokens via self-attention. The distillation token is trained to mimic the teacher's soft labels, enabling knowledge transfer without modifying the transformer architecture.
• Teacher-Student Framework: Uses a CNN (e.g., RegNetY) as the teacher, which provides richer soft targets than hard labels. The student transformer learns from both the ground-truth labels and the teacher's predictions, with a distillation loss that combines cross-entropy and KL divergence.
• Hard Distillation: The distillation token can also be trained with hard labels from the teacher's argmax prediction, which simplifies training and yields similar performance.
• Efficient Training: DeiT-Base is trained on 4 GPUs for 3 days, demonstrating practical feasibility.

Results

• DeiT-Base achieves 83.1% top-1 accuracy on ImageNet, matching the performance of EfficientNet-B5 (83.3%) without convolutions.
• DeiT-Small and DeiT-Tiny variants achieve 79.9% and 72.2% respectively, showing scalability.
• The distillation token alone contributes a 1.5% improvement over the baseline without distillation.
• Training with a CNN teacher (RegNetY-16GF) yields better results than using a transformer teacher, suggesting that complementary inductive biases help.

Significance

DeiT bridges the gap between CNNs and transformers in vision by showing that pure attention models can be trained efficiently on standard datasets. It popularized the use of distillation tokens, which have since been adopted in many subsequent vision transformer works. The paper also influenced the development of data-efficient transformers and inspired further research into hybrid architectures. For practitioners, DeiT provides a practical recipe for deploying transformers in image classification without requiring massive compute or data resources.