Gecko

Analysis

Why This Paper Matters

Text embeddings are the backbone of modern search, question answering, and semantic similarity systems. While large language models (LLMs) have pushed the boundaries of what's possible, their size and cost make them impractical for many real-time applications. Gecko addresses this gap by distilling the knowledge of massive LLMs into a compact 1.2B parameter model that achieves state-of-the-art performance on the MTEB benchmark, often matching or exceeding models with 7B+ parameters. This makes high-quality embeddings accessible for deployment at scale.

How It Works

Gecko's training pipeline consists of two main stages: pre-fine-tuning and fine-tuning. The key innovation is the creation of FRet (Few-shot Prompted Retrieval dataset), a diverse set of 6.6M query-passage pairs generated through a two-step LLM distillation process.

Pre-Fine-Tuning: The model is first trained on large-scale community QA and web title-body pairs using contrastive learning. A task-specific token is prepended to each query to inform the model about the type of retrieval task. The objective is a standard in-batch cross-entropy loss with in-batch negatives.

FRet Generation: The core of Gecko's success lies in how it generates training data. An LLM is prompted to read a random web passage and produce both a task description (e.g., "question answering") and a relevant query. This process is repeated across diverse web passages, yielding a wide variety of tasks and styles.

For each generated query, an existing embedding model retrieves the top N passages from the corpus. The same LLM then ranks these passages using two complementary methods: query likelihood (measuring how likely the query is given the passage) and relevance classification (measuring the likelihood of a relevance label). The rankings are combined via Reciprocal Rank Fusion (RRF), and the top-ranked passage becomes the positive target, while low-ranked passages serve as hard negatives.

Fine-Tuning: The final model is fine-tuned on a mixture of FRet and academic datasets (Natural Questions, HotpotQA, FEVER, etc.) using the same contrastive loss. To support multiple embedding dimensions (768 and 256), Gecko employs a Matryoshka Representation Learning (MRL) loss, which optimizes sub-dimensions of the embedding vector.

Results

Gecko-1b-768 achieves an average MTEB score of 66.31, outperforming all models under 5B parameters. It matches or exceeds larger models like text-embedding-3-large (66.31 vs. 64.59) and E5-mistral (66.31 vs. 66.23). On specific tasks, Gecko sets new state-of-the-art results on classification, STS, and summarization. Even when trained solely on FRet (zero-shot on MTEB), Gecko performs competitively.

In multilingual settings, Gecko trained with additional MIRACL data outperforms all baselines on the MTEB benchmark. Remarkably, the model trained only on English FRet data still achieves superior multilingual performance, suggesting that the distillation process captures language-agnostic retrieval patterns.

Significance

Gecko demonstrates that careful data generation and distillation can produce compact models that rival much larger ones. This has practical implications for deploying high-quality embeddings in resource-constrained environments, such as mobile devices or real-time search systems. The success of English-only training for multilingual tasks also hints at the possibility of building universal embedding models with minimal language-specific data. Future work could explore extending this approach to even smaller models or incorporating more diverse task types.