Jina Bilingual Embeddings

Analysis

Why This Paper Matters

Multilingual text embeddings are critical for cross-lingual search, retrieval, and classification, but most existing models (e.g., mBERT, XLM-R) are designed to handle many languages simultaneously, often sacrificing per-language performance for breadth. Jina Bilingual Embeddings challenge this trade-off by training dedicated models for specific language pairs (English-German and English-Spanish). The key insight: by focusing on just two languages, the model can allocate more capacity to each, achieving higher accuracy on both languages while supporting long contexts up to 8192 tokens. This is especially valuable for enterprise applications like legal document retrieval or customer support where a single language pair dominates.

How It Works

The training pipeline follows a three-stage approach similar to Jina Embeddings v2, but tailored for bilingual settings.

Stage 1: Pre-training a Modified BERT

The backbone is a BERT architecture modified with ALiBi (Attention with Linear Biases) to handle long sequences without position embeddings. For the Spanish model, key-query normalization is added to stabilize training. Tokenizers use a doubled vocabulary (compared to monolingual) to accommodate both languages. Pre-training data includes CulturaX, Wikipedia, and Opus, totaling 256B tokens per model (half English, half target language).

Stage 2: Fine-tuning with Text Pairs

Following Sentence-BERT, a mean pooling layer is added. The model is trained on 518M English, 211M German, and 111M Spanish text pairs (97.5% monolingual, 2.5% parallel). Data sources include MQA, XL-Sum, XNLI, and Common Crawl (title-content and FAQ pairs). A two-stage quality filter removes short/long texts and repetitive lines, then applies near-deduplication. The loss is bi-directional InfoNCE:

Stage 3: Multi-Task Fine-tuning

To handle diverse downstream tasks, the model is fine-tuned with a multi-task objective. For retrieval tasks (e.g., MSMarco, Natural Questions), a bi-directional InfoNCE loss with one positive and multiple negatives is used:

For STS tasks, negative Pearson correlation is used:

Since high-quality English data is abundant, machine translation (WMT19 for German, MADLAD-3B for Spanish) is used to create target-language datasets. Additional German datasets (GerDaLIR, GermanDPR) and Spanish datasets (SQAC, XL-Sum subset, MIRACL) are included.

Results

Backbone Model Performance

On GLUE and XTREME benchmarks, bilingual models outperform multilingual baselines (mBERT, XLM-RoBERTa) on average. The Spanish model achieves the highest GLUE score, while the German model slightly edges XLM-R on XTREME. Notably, bilingual models show a significant advantage in zero-shot cross-lingual transfer on BUCC and Tatoeba sentence retrieval tasks.

Embedding Model Evaluation

On MTEB tasks (extended for German and Spanish), the Spanish model outperforms multilingual-e5 on clustering, retrieval, and reranking, while slightly underperforming in classification. The German model beats multilingual-e5 on retrieval and STS, with comparable performance elsewhere. Cross-lingual German-English tasks show parity or improvement over multilingual-e5.

Ablation: Bilingual vs. Multilingual

When fine-tuned on a small set of German-English pairs, bilingual models (including BiBERT) outperform multilingual models on German and cross-lingual tasks, and slightly edge them on English tasks.

Multi-Task Learning

The Pearson variant (using Lcorr loss for STS) outperforms MSE and no-STS variants on German and Spanish STS tasks, especially on in-domain datasets (SICK-R, STS12).

Significance

Jina Bilingual Embeddings demonstrate that dedicated bilingual models can outperform general multilingual models on both languages, while maintaining efficiency on English tasks. This is a practical win for applications where one language pair dominates, such as German-English legal search or Spanish-English customer support. The open-source release on HuggingFace lowers the barrier for adoption. Future work could extend this approach to other language pairs and explore scaling to longer contexts beyond 8192 tokens.