SigLip

Analysis

Why This Paper Matters

Contrastive learning has been the backbone of vision-language models like CLIP, but its reliance on softmax normalization across the batch creates a fundamental bottleneck: the loss function requires a global view of all pairwise similarities, which limits batch size scaling and introduces asymmetry between image and text directions. SigLip challenges this paradigm by proposing a simple sigmoid loss that treats each image-text pair independently as a binary classification problem. This shift not only simplifies the training pipeline but also unlocks better performance at smaller batch sizes—a critical advantage for practitioners with limited computational resources. Moreover, it allows scaling to extremely large batch sizes (up to 1M) without the performance degradation seen with softmax loss. The paper demonstrates that this approach achieves state-of-the-art results on zero-shot ImageNet classification and multilingual retrieval tasks, often surpassing much larger models.

How It Works

The core idea is to replace the standard softmax-based contrastive loss with a pairwise sigmoid loss. In traditional contrastive learning (e.g., CLIP), for a mini-batch of N image-text pairs, the loss normalizes similarities across all pairs using softmax, effectively computing two separate cross-entropy losses: one for images and one for texts. This requires computing the full N×N similarity matrix and normalizing each row/column.

In contrast, the sigmoid loss processes each pair (i,j) independently. For a given image I_i and text T_j, the loss is:

where z_ij = +1 if i=j (positive pair) and -1 otherwise (negative pair). This formulation turns the problem into a binary classification over all N² pairs. However, because negatives vastly outnumber positives (N² - N vs N), the loss is initially dominated by negatives, causing large optimization steps. To mitigate this, the authors introduce a learnable bias term b (similar to the temperature t):

This bias helps balance the positive/negative contributions at initialization. The sigmoid loss is also symmetric (unlike softmax), treating image-to-text and text-to-image directions equally.

Results

The authors conducted extensive experiments across batch sizes from 512 to 1M. Key findings:

• For batch sizes below 16k, sigmoid loss significantly outperforms softmax loss.
• As batch size increases, softmax loss improves but never surpasses sigmoid loss; at very large batch sizes (e.g., 1M), sigmoid still matches or slightly beats softmax.
• Optimal batch size for sigmoid loss is around 32k; beyond that, gains are marginal.

In the SigLiT experiment (using precomputed image embeddings from a ViT-g model and training only the text tower), the best model achieved 84.7% zero-shot ImageNet accuracy with a base-sized text model, compared to 85.2% from the original LiT paper which used a much larger g-sized text model.

For multilingual pre-training (mSigLIP), the model was trained on WebLI covering 100 languages. Using a bottlenecked token embedding (compressing a 250k vocabulary to an intermediate size K before mapping to full dimension W=768) achieved nearly the same performance as a full large vocabulary, with only ~0.5% drop on ImageNet zero-shot when K=96. The multilingual model set a new state-of-the-art on XM3600 text-to-image retrieval: 34.9% with a base model, surpassing the previous 28.5% from a 4B-parameter ViT-e model.

When scaling up to 40B examples with larger image patches (256) and text tokens (64), the SigLIP ViT-B model achieved 42.6% image retrieval recall@1 and 54.1% text retrieval recall@1 on XM36000 across 36 languages, again state-of-the-art for a base model.

Significance

SigLip's contribution is both practical and conceptual. Practically, it enables training high-quality vision-language models with smaller batch sizes, democratizing access to this technology. Conceptually, it challenges the necessity of global normalization in contrastive learning, showing that a simple per-pair binary classification can be more effective. The ability to scale to extremely large batch sizes without performance loss also opens the door to training on massive datasets with efficient hardware utilization. The multilingual results are particularly impressive, demonstrating that a base-sized model can outperform models orders of magnitude larger. This work will likely influence future designs of contrastive learning objectives across modalities.