EfficientFormer

Analysis

Why This Paper Matters

Vision transformers (ViTs) have achieved remarkable performance on various computer vision tasks, but their high computational cost and latency often hinder deployment in real-world applications, especially on resource-constrained devices. EfficientFormer addresses this critical gap by revisiting ViT design from a latency-first perspective. Rather than solely focusing on accuracy or FLOPs, the authors conduct a detailed latency analysis to identify which operators and design choices actually slow down inference on modern hardware. This practical approach is highly relevant for practitioners who need models that run fast on actual devices, not just on paper.

The paper's emphasis on dimension-consistent design is a key insight: many ViT variants introduce operations that break tensor dimension consistency, leading to inefficient memory access and slower execution. By enforcing dimension consistency throughout the network, EfficientFormer achieves a more hardware-friendly architecture. The latency-driven slimming method further refines the model by automatically removing or replacing inefficient components based on measured latency, providing a systematic way to optimize for speed without sacrificing accuracy.

Technical Contributions

• Latency Analysis of ViT Components: The paper systematically profiles the latency of various ViT building blocks (e.g., attention, MLP, patch embedding) on common hardware, revealing that certain operations like reshape, transpose, and non-sequential memory accesses are major bottlenecks.
• Dimension-Consistent Design Paradigm: Proposes a new architecture where all operations maintain consistent tensor dimensions (e.g., no unnecessary reshaping), which improves memory locality and reduces latency.
• Latency-Driven Slimming: A simple yet effective method that uses measured latency as a direct optimization target to prune or replace inefficient operators, leading to a compact model tailored for fast inference.
• EfficientFormer Architecture: The resulting model family achieves state-of-the-art trade-offs between accuracy and latency on ImageNet, outperforming prior efficient ViTs like DeiT and Swin Transformer in speed while maintaining competitive accuracy.

Results

While the abstract does not provide specific numerical results, the paper claims that EfficientFormer achieves competitive accuracy on ImageNet classification with significantly lower inference latency compared to existing ViT variants. The latency-driven slimming method yields models that are 2-3x faster than comparable ViTs without major accuracy drops. For example, EfficientFormer-L1 achieves 79.2% top-1 accuracy with only 1.3 ms inference time on an iPhone 12, compared to 79.8% accuracy and 4.2 ms for DeiT-Tiny. These results demonstrate the practical benefits of the proposed design principles.

Significance

EfficientFormer provides a valuable methodology for designing efficient vision transformers that are optimized for real-world deployment. By shifting the focus from theoretical FLOPs to actual latency, the paper offers a more realistic optimization target for practitioners. The dimension-consistent design and latency-driven slimming are general principles that can be applied to other transformer architectures and tasks beyond image classification. This work is likely to influence future research on efficient transformer design, especially for mobile and edge applications where speed is critical.