Jina Embeddings v2

Analysis

Why This Paper Matters

Most open-source text embedding models struggle with long documents. They truncate input, forcing users to split texts into small chunks. This multiplies the number of vectors, increasing memory use and slowing down vector search. Jina Embeddings v2 directly tackles this by supporting up to 8192 tokens in a single embedding, eliminating the need for chunking. This is a practical breakthrough for applications like legal document analysis, scientific paper retrieval, and long-form question answering.

How It Works

The authors build on BERT but introduce key architectural changes to handle long sequences. First, they replace absolute positional embeddings with ALiBi (Attention with Linear Biases). ALiBi adds a constant bias to attention scores based on token distance, allowing the model to extrapolate to longer sequences than seen during training. For the encoder, they use a symmetric version of ALiBi that works with bidirectional attention.

Second, they replace standard feedforward layers with gated linear units (GLU). The small and base models use GEGLU (GELU activation), while the large model uses ReGLU (ReLU activation). They keep post-layer normalization, as pre-layer normalization did not improve stability.

Training happens in three stages:

1. Pre-training from scratch on the C4 dataset (170B tokens). They mask 30% of tokens using whole-word masking and train with standard MLM loss. Because ALiBi doesn't need learned position embeddings, they pre-train on 512-token sequences and later adapt to longer ones.

2. Fine-tuning on text pairs using InfoNCE loss. They use ~40 diverse data sources to pull similar texts together and push dissimilar ones apart in embedding space.

1. Fine-tuning with hard negatives to improve ranking. Each batch includes one positive and 15 negative examples. For retrieval datasets, hard negatives come from similar passages; for others, negatives are random. They use a modified InfoNCE loss (LNCE+) that incorporates all negatives.

Results

JinaBERT models achieve competitive scores on the GLUE benchmark, matching or exceeding standard BERT variants. More importantly, they maintain high MLM accuracy even when extrapolated to 8192 tokens, while BERT and RoBERTa degrade sharply.

On the MTEB benchmark, Jina-base-v2 (137M params) outperforms jina-small-v2 (33M) on classification and retrieval tasks. Both models beat smaller models like all-MiniLM-L6-v2, especially on retrieval, thanks to the second-stage hard-negative fine-tuning.

On long-text tasks, longer sequences improve performance for NarrativeQA retrieval and BigPatent clustering. However, for WikiCities clustering, longer sequences slightly hurt because the first few sentences already contain the country information.

Significance

Jina Embeddings v2 is a practical, open-source solution for long-document embeddings. It removes the need for chunking, reducing memory and latency in production systems. The architectural choices—ALiBi and GLU—are simple yet effective, and the three-stage training pipeline is reproducible. This work pushes the field toward more capable and efficient text embeddings for real-world applications.