Faster RCNN

Analysis

Why This Paper Matters

Before Faster R-CNN, object detection systems were slow and fragmented. The original R-CNN used selective search to generate region proposals, then classified each proposal with a CNN — a process that took seconds per image. Fast R-CNN improved speed by sharing convolutional computation across proposals, but still relied on external proposal methods. Faster R-CNN eliminated this bottleneck by integrating proposal generation directly into the network, achieving near real-time performance without sacrificing accuracy. This paper is a cornerstone of modern object detection, demonstrating that end-to-end learning of both proposal and detection is not only possible but highly effective.

How It Works

Faster R-CNN is composed of two main modules: a Region Proposal Network (RPN) and a Fast R-CNN detector. Both share the same convolutional feature extractor (e.g., VGG-16). The RPN takes the feature map from the last shared convolutional layer and slides a small network over it. At each sliding window location, the network outputs multiple region proposals using a set of reference boxes called anchors.

The RPN is a fully convolutional network. For each sliding window position, it produces two outputs: a box-regression layer (reg) that refines the coordinates of k proposals, and a box-classification layer (cls) that estimates whether each proposal contains an object or not. The k proposals are parameterized relative to k anchor boxes, each with a specific scale and aspect ratio. By default, 3 scales and 3 aspect ratios are used, yielding k=9 anchors per location. For a feature map of size W×H, this results in WHk total anchors.

Anchors are a key innovation. They allow the RPN to predict proposals of varying sizes and shapes without needing multiple image scales or pyramid processing. Each anchor is centered at the sliding window location and comes with predefined scale and aspect ratio. During training, anchors are assigned binary labels: positive if they have the highest IoU with a ground-truth box or an IoU > 0.7 with any ground-truth box; negative if IoU < 0.3. The loss function for RPN combines classification loss (binary cross-entropy) and regression loss (smooth L1) for the positive anchors.

To share features between RPN and Fast R-CNN, the authors used an alternating training strategy: first train RPN, then use its proposals to train Fast R-CNN, then re-initialize RPN with the tuned network, and iterate. This approach ensures that both modules converge to a shared representation.

Results

On the PASCAL VOC 2007 benchmark, Faster R-CNN achieved 73.2% mean average precision (mAP) with VGG-16, outperforming Fast R-CNN (70.0% mAP) and R-CNN (66.0% mAP). More importantly, it ran at 5 frames per second (fps) on a GPU, compared to 0.5 fps for Fast R-CNN and 0.02 fps for R-CNN. This represented a tenfold speedup over Fast R-CNN while improving accuracy. The RPN itself added only a small computational overhead (about 10 ms per image), making the entire pipeline practical for real-time applications.

Significance

Faster R-CNN fundamentally changed how object detection systems are designed. By making region proposal generation a learnable part of the network, it eliminated the last major handcrafted component in the detection pipeline. This work directly inspired subsequent architectures like Mask R-CNN (instance segmentation), Feature Pyramid Networks (multi-scale detection), and even one-stage detectors like RetinaNet. The concept of anchor boxes became a standard tool in object detection, adopted by YOLO, SSD, and many others. Faster R-CNN remains a strong baseline and is widely used in industry and research, proving that careful integration of proposal and detection can achieve both speed and accuracy.