Conference Paper
Computer Vision
Featured

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Liang-Chieh Chen(Google (United States)), George Papandreou(Google (United States)), Iasonas Kokkinos(University College London), Kevin Murphy(Google (United States)), Alan Yuille(Johns Hopkins University)
April 27, 2017IEEE Transactions on Pattern Analysis and Machine Intelligence22,292 citations
arXivDOI

22k

Citations

1.9k

Influential Citations

IEEE Transactions on Pattern Analysis and Machine Intelligence

Venue

2017

Year

Abstract

In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First, we highlight convolution with upsampled filters, or 'atrous convolution', as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second, we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third, we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed "DeepLab" system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.

Analysis

Why This Paper Matters

DeepLab represents a pivotal moment in semantic image segmentation, bridging the gap between deep convolutional networks and structured prediction. Before DeepLab, segmentation models often struggled with two fundamental issues: the loss of spatial resolution due to repeated downsampling, and the inability to incorporate multi-scale context without exploding parameters. By introducing atrous convolution and ASPP, the paper directly addressed these bottlenecks, enabling both high-resolution feature maps and robust multi-scale reasoning. The integration of a fully connected CRF further demonstrated that deep learning and probabilistic graphical models could be combined to refine object boundaries, a problem that pure DCNNs had not fully solved.

This work set a new state-of-the-art on PASCAL VOC-2012, a benchmark that defined the field for years, and its influence extended to later architectures like DeepLabv3 and DeepLabv3+. The paper's emphasis on practical, reproducible contributions—with publicly released code—made it a go-to reference for both academic researchers and industry practitioners. Its ideas have been widely adopted in autonomous driving, medical imaging, and satellite imagery analysis, where precise segmentation is critical.

Technical Contributions

  • Atrous Convolution: Allows explicit control of feature resolution by inserting zeros between filter weights, effectively enlarging the field-of-view without increasing parameters or computation. This enables dense feature maps at original image resolution.
  • Atrous Spatial Pyramid Pooling (ASPP): Probes the convolutional feature layer with multiple atrous rates (e.g., 6, 12, 18, 24) to capture objects and context at multiple scales, then concatenates the responses for robust segmentation.
  • Fully Connected CRF: Applied as a post-processing step to refine object boundaries. Unlike traditional CRFs with short-range connections, this uses all pairwise potentials, leveraging the DCNN's unary scores and edge-preserving bilateral filters to improve localization.
  • Combined Training: The DCNN is trained end-to-end with atrous convolution, while the CRF parameters are learned separately, keeping the system modular and efficient.

Results

On the PASCAL VOC-2012 test set, DeepLab achieved 79.7% mean Intersection-over-Union (mIOU), surpassing prior methods by a significant margin. On PASCAL-Context, it reached 45.7% mIOU; on PASCAL-Person-Part, 64.9%; and on Cityscapes, 70.4%. These results were obtained with a single model and without any external data, highlighting the effectiveness of the proposed components. Ablation studies confirmed that each contribution—atrous convolution, ASPP, and CRF—provided measurable gains, with the full system outperforming baselines by over 5% mIOU.

Significance

DeepLab's innovations directly influenced the design of later segmentation networks, including DeepLabv3 (which replaced CRF with improved ASPP) and DeepLabv3+ (which added encoder-decoder structure). The concept of atrous convolution has been adopted in other dense prediction tasks like depth estimation and optical flow. By open-sourcing the code, the authors enabled rapid iteration and benchmarking, accelerating progress in the field. The paper remains one of the most cited in computer vision, with over 22,000 citations, and its techniques are now standard in modern segmentation pipelines.

semantic-segmentationdeep-learningcomputer-visionatrous-convolutionasppconditional-random-fieldspascal-voc
Neural Networks

Related Papers