BEiT

Analysis

Why This Paper Matters

Before BEiT, applying BERT-style pretraining to images was tricky. Language has a clear vocabulary (words, subwords) that makes masked prediction straightforward. Images, on the other hand, are continuous pixel grids. Early attempts either treated masked patch prediction as regression to raw pixels, which wasted capacity on high-frequency noise and short-range textures, or relied on contrastive learning (like MoCo, SimCLR) that required careful negative sampling. BEiT elegantly solves this by introducing a discrete visual vocabulary, enabling a direct analog of masked language modeling for vision. This opened the door for a family of masked image modeling methods (MAE, MaskFeat, etc.) and showed that generative pretraining could rival contrastive approaches.

How It Works

BEiT operates on two distinct representations of an image: image patches (the input) and visual tokens (the prediction targets).

Image Patches as Input: Following ViT, the image is divided into a grid of non-overlapping patches (e.g., 14×14 patches of size 16×16 for a 224×224 image). Each patch is flattened and linearly projected into an embedding, similar to word embeddings in BERT. A special [S] token is prepended, and standard learnable 1D position embeddings are added. This sequence is fed into a standard Transformer encoder.

Visual Tokens as Targets: To create a discrete prediction space, BEiT uses a pretrained discrete variational autoencoder (dVAE) to tokenize the image into a grid of visual tokens (also 14×14). The dVAE has a codebook of 8192 visual tokens. The tokenizer maps each image region to a token index; the decoder reconstructs the image from these tokens. During BEiT pretraining, these tokens serve as the ground-truth labels for masked patches.

Masked Image Modeling (MIM): During pretraining, roughly 40% of image patches are randomly masked (their embeddings are replaced with a learnable mask embedding). The Transformer processes the full sequence (masked + unmasked patches) and outputs representations for each position. For each masked position, a linear classifier (with softmax over the 8192 token vocabulary) predicts the visual token that the dVAE assigned to that patch. The loss is cross-entropy between predicted and true token indices. This is identical in spirit to BERT's masked language model.

The backbone is a standard ViT architecture, so results are directly comparable to prior work. Pretraining uses only ImageNet-1K images (no labels) with standard augmentations (random crop, flip, color jitter).

Results

BEiT is evaluated on ImageNet-1K classification (1.3M images, 1000 classes) and ADE20K semantic segmentation (25K images, 150 categories). For classification, a simple linear classifier on top of the [S] token's representation achieves competitive accuracy, demonstrating that the pretrained features are semantically rich. For segmentation, BEiT serves as a backbone encoder, with deconvolution layers as decoder; it obtains strong mIoU scores, showing good transfer to dense prediction tasks. The results confirm that masked image modeling with discrete tokens is an effective self-supervised pretraining strategy for vision Transformers.

Significance

BEiT's core insight—using a discrete tokenizer to create a visual vocabulary—was transformative. It showed that BERT-style pretraining could work for images without sacrificing performance to pixel-level noise. This inspired a wave of follow-up works (MAE, MaskFeat, SimMIM) that further simplified or improved the approach. BEiT also unified the pretraining paradigm across modalities, making it easier to transfer ideas from NLP to vision. Its impact extends beyond classification to segmentation, detection, and video understanding, cementing masked image modeling as a cornerstone of modern vision self-supervised learning.