NeoBERT

Analysis

Why This Paper Matters

The era of massive language models often overshadows the humble encoder. Yet, for tasks like classification, retrieval, and embedding, bidirectional encoders remain indispensable. NeoBERT arrives as a breath of fresh air, proving that thoughtful architectural and data choices can yield a compact 250M-parameter model that rivals or surpasses much larger counterparts. Its fully open-source nature—code, data, training scripts, and checkpoints—makes it a practical gift for the AI community, lowering barriers to high-quality NLP.

How It Works

NeoBERT builds on the stable Transformer foundation but injects several modern tweaks. The core insight is an optimal depth-to-width ratio: instead of widening the model, it deepens it to 28 layers while keeping the hidden size at 768, maximizing parameter efficiency. This is illustrated in the architecture comparison:

Positional information uses Rotary Position Embeddings (RoPE), compatible with YaRN for extended contexts. Layer normalization swaps to RMSNorm for computational savings, and activations use SwiGLU with a 2/3 hidden-size scaling to maintain parameter count.

Data is a game-changer. NeoBERT trains on RefinedWeb, a 600B-token dataset from filtered Common Crawl—18 times larger than RoBERTa's corpus. Pre-training follows a two-stage procedure: first 1M steps (2T tokens) on sequences up to 1,024 tokens, then 50k steps (100B tokens) on sequences up to 4,096 tokens. The masking rate is increased to 20% based on recent findings.

Ablation studies (M0-M10) reveal key insights:

• Data size (RefinedWeb) gave a +3.6% boost.
• Model size increase to 250M added +2.9%.
• Tokenizer switch to LLaMA BPE hurt (-2.1%), likely due to multilingual trade-offs.
• Packed sequences without cross-attention caused a -2.8% drop.
• AdamW with cosine decay and 20% masking initially dropped performance but were retained for expected long-training benefits.

Results

NeoBERT achieves a GLUE score of 89.0, competitive with models 100-150M parameters larger:

On MTEB, after 2,000 steps of contrastive fine-tuning, NeoBERT leads with a +4.5% relative improvement over the second-best model:

Long-sequence handling is impressive: NeoBERT1024 generalizes to ~3,000 tokens despite training on 1,024, and NeoBERT4096 extends further with additional training:

Inference efficiency shines for long sequences: NeoBERT is 46.7% faster than ModernBERTbase at 4,096 tokens:

Significance

NeoBERT demonstrates that encoder models are far from obsolete. By carefully balancing depth, data, and training strategy, it achieves state-of-the-art results with a fraction of the parameters. Its open-source release empowers practitioners to fine-tune for custom tasks without prohibitive compute. The two-stage context extension offers a compute-efficient path to longer sequences, crucial for retrieval-augmented generation and document understanding. NeoBERT sets a new benchmark for what a compact, efficient encoder can achieve.