Model quantization and hardware acceleration for vision transformers: A comprehensive survey

Analysis

Why This Paper Matters

Vision transformers (ViTs) have achieved state-of-the-art performance in many computer vision tasks, but their high computational cost and memory footprint hinder deployment on edge devices. This comprehensive survey addresses the critical need for efficient ViT inference by reviewing model quantization and hardware acceleration techniques. As AI moves toward on-device processing, understanding how to compress ViTs without significant accuracy loss is essential for real-world applications like autonomous driving, robotics, and mobile vision.

The paper is timely given the rapid adoption of ViTs and the growing emphasis on sustainable AI. By consolidating disparate research threads, it provides a structured overview that helps practitioners navigate the complex landscape of quantization methods and hardware support.

Technical Contributions

The survey makes several key contributions:

• Taxonomy of quantization methods: Categorizes approaches into post-training quantization (PTQ) and quantization-aware training (QAT), further subdivided by granularity (per-tensor, per-channel, per-vector) and bit-width (binary, ternary, low-bit).
• Hardware acceleration review: Discusses specialized hardware (e.g., FPGAs, ASICs, GPU optimizations) for executing quantized ViTs, including support for mixed-precision and non-uniform quantization.
• Cross-layer optimization: Highlights techniques that jointly optimize quantization and hardware mapping to minimize latency and energy.
• Benchmarking and datasets: Summarizes common benchmarks (e.g., ImageNet, COCO) and metrics (accuracy, throughput, energy) used to evaluate quantized ViTs.

Results

As a survey, the paper does not present new experimental results. Instead, it aggregates findings from prior works, reporting that post-training quantization can reduce model size by 4× with less than 1% accuracy drop on ImageNet for ViT-B/16. Quantization-aware training often achieves better accuracy at extreme low-bit widths (e.g., 4-bit) but requires more training time. Hardware acceleration using FPGA implementations can achieve up to 10× speedup over GPU baselines for quantized ViTs.

Significance

This survey provides a foundational reference for the efficient deployment of vision transformers, a critical step toward democratizing advanced AI capabilities on edge devices. By systematically organizing existing techniques, it helps researchers identify gaps—such as the lack of standardized benchmarks for quantized ViTs and limited support for dynamic quantization in hardware. The paper's impact lies in accelerating progress toward practical, low-power vision systems that can operate in real-time without cloud dependency.