ModernBERT

Analysis

Why This Paper Matters

Encoder-only models like BERT have been foundational for NLP tasks such as classification, retrieval, and question answering. However, they have seen less architectural innovation compared to decoder-only models in recent years. ModernBERT addresses this gap by introducing a suite of modern techniques—GeGLU activations, RoPE embeddings, alternating attention, and hardware-aware design—to create a more efficient and performant encoder. Its ability to handle long contexts (8192 tokens natively) and code data makes it particularly relevant for retrieval-augmented generation (RAG) and code search applications. The paper demonstrates that encoder-only models can still achieve Pareto improvements over older architectures, offering practitioners a drop-in replacement that is faster and more memory-efficient.

How It Works

ModernBERT is a deep-and-narrow transformer with 22 or 28 layers (base and large versions). Key architectural changes include:

• Bias removal: Bias terms are disabled in all linear layers except the final decoder, and in LayerNorms, allowing more parameters to be used in weight matrices.
• Positional embeddings: Rotary Positional Embeddings (RoPE) replace absolute positional embeddings, enabling better generalization to long sequences and easier context extension.
• Normalization: Pre-normalization with standard LayerNorm is used, with an extra LayerNorm after the embedding layer but the first attention LayerNorm removed to avoid redundancy.
• Activation: GeGLU (Gated Linear Unit with GeLU) replaces the original GeLU activation, improving gradient flow and expressivity.

Efficiency improvements are central to ModernBERT:

• Alternating attention: Every third layer uses global attention (all tokens attend to all others) with a high RoPE theta (160,000), while the remaining layers use local sliding window attention (128 tokens) with a lower theta (10,000). This reduces computational cost while preserving long-range dependencies.
• Unpadding: Padding tokens are removed before the embedding layer, and sequences in a minibatch are concatenated into a single sequence. This avoids wasted compute on padding tokens and yields 10–20% performance improvement over prior unpadding methods.
• Flash Attention: Uses Flash Attention 3 for global layers and Flash Attention 2 for local layers, providing memory-efficient attention kernels.
• torch.compile: PyTorch's compiler is used to optimize training throughput by 10% with minimal overhead.

The model design balances depth and width for hardware efficiency. Base model has hidden size 768 with GLU expansion 2304, while large has hidden size 1024 with expansion 5248. These ratios are chosen for optimal tiling on GPU tensor cores.

Training uses a modified OLMo tokenizer with vocabulary size 50,368 (multiple of 64) and includes 83 unused tokens for downstream tasks. The data mixture includes web documents, code, and scientific literature (2 trillion tokens). Training follows a three-phase schedule: 1.7 trillion tokens at 1024 sequence length with RoPE theta 10,000, then context extension to 8192 tokens by increasing global attention RoPE theta to 160,000 for 300 billion tokens, and finally a decay phase with upsampled high-quality sources. Masked language modeling uses a 30% masking rate (higher than the original 15%) and no next-sentence prediction.

Results

ModernBERT achieves state-of-the-art results across multiple benchmarks:

• GLUE: Outperforms DeBERTa-v3-base and is near state-of-the-art for large models.
• BEIR: Superior performance in both single-vector (DPR) and multi-vector (ColBERT) retrieval settings.
• MLDR (long-context retrieval): Strong performance, especially in multi-vector setting; single-vector results suggest potential for long-context fine-tuning.
• Code retrieval: Excels on CodeSearchNet and StackOverflow-QA, likely due to code-inclusive pretraining.

Efficiency benchmarks on an NVIDIA RTX 4090 show:

• ModernBERT processes long documents 2.65–3x faster than competitors at base and large sizes.
• At 8192 tokens, ModernBERT-large's speed is closer to GTE-en-MLM base than GTE-en-MLM-large.
• ModernBERT supports larger batch sizes (better memory efficiency) than all compared models.

Significance

ModernBERT represents a significant step forward for encoder-only models, demonstrating that careful architectural modernization can yield substantial gains in both performance and efficiency. For practitioners, it offers a practical upgrade path from older BERT variants, especially for tasks requiring long-context understanding or code processing. The model's efficiency improvements—particularly unpadding and alternating attention—make it suitable for deployment in latency-sensitive applications like real-time search and RAG pipelines. The open-source release on HuggingFace further lowers the barrier for adoption. Future work could explore fine-tuning strategies for specific domains or extending the approach to multilingual settings.