Tulu v2

Analysis

Why This Paper Matters

The landscape of open-source instruction-tuned language models has evolved rapidly, yet the community still lacks systematic guidelines on how to combine data sources and training techniques for maximum performance. Tulu 2 addresses this gap by releasing a carefully curated instruction mixture (V2 mix) and showing how to scale direct preference optimization (DPO) to 70B parameters. This work is particularly valuable because it provides reproducible recipes—data, training code, and model weights—that allow practitioners to build competitive models without relying on proprietary APIs. The paper also explores practical trade-offs: whether parameter-efficient fine-tuning (QLoRA) can match full fine-tuning, and how domain-specific base models (Code LLAMA) can outperform larger general models on coding tasks.

How It Works

The core contribution is the TÜLU-V2-mix, a filtered collection of 326,154 instruction examples drawn from 11 sources. The authors deliberately include chain-of-thought data (CoT), distilled GPT-4 outputs (GPT4-Alpaca, WizardLM, Open-Orca), and high-quality human annotations (Open Assistant, LIMA). They also add a small set of hardcoded prompts to ensure the model can correctly answer questions about its identity. Compared to the V1 mixture (490k samples), V2 is smaller but more diverse, and the context length is extended from 2,048 to 8,192 tokens.

For training, the authors fine-tune all sizes of LLAMA-2 (7B, 13B, 70B) and Code LLAMA (7B, 13B, 34B) on the V2 mix. They then apply DPO following the Zephyr-Beta recipe: training on filtered, binarized UltraFeedback data for three epochs with a low learning rate (5e-7). DPO is chosen for its simplicity—it avoids the complexity of reinforcement learning while achieving similar alignment benefits. The authors also test QLoRA during instruction tuning to see if memory requirements can be reduced, but find it underperforms full fine-tuning on open-ended generation tasks (AlpacaEval), though the gap shrinks with larger models.

Results

TÜLU 2 70B achieves the highest average score across seven benchmarks among open-source models, and is competitive with GPT-3.5 (March 2023 version). On three of seven tasks it is the top open model, and on the remaining four it is within 1% of the best. However, a clear gap remains with GPT-4 and GPT-3.5-turbo.

Comparing V1 and V2 mixtures, models trained on V2 generally outperform those on V1 on BBH, Codex-Eval, AlpacaEval, and TruthfulQA, but lag slightly on GSM8k and TydiQA, suggesting a trade-off in multilingual and math reasoning. The V2 mix also consistently beats training on ShareGPT alone.

DPO training yields large gains on AlpacaEval and MT-Bench, especially for the 13B and 70B models. TÜLU 2+DPO 70B becomes the top open model on MT-Bench and second-best on AlpacaEval. Importantly, DPO does not degrade MMLU, BBH, or GSM8k, but it does reduce TydiQA performance, likely because UltraFeedback lacks multilingual data. DPO also increases verbosity, though less than other open models.

QLoRA underperforms full fine-tuning on AlpacaEval, but the gap narrows as model size increases, hinting that at very large scales QLoRA might become viable for instruction tuning.

CODE TÜLU 2 models dramatically outperform TÜLU 2 on coding benchmarks—the 7B Code model matches the 70B general model on Codex-Eval. However, they suffer a ~20% drop on AlpacaEval, indicating a trade-off between coding and general instruction following.

Significance

Tulu 2 provides a practical blueprint for building capable open-source instruction-tuned models. The V2 mix demonstrates that careful data curation—balancing quality, diversity, and domain coverage—can yield strong results even with fewer examples. The successful scaling of DPO to 70B parameters shows that alignment techniques can be applied at large scale without destabilizing training, and the resulting model is competitive with proprietary alternatives. The analysis of QLoRA and Code LLAMA offers actionable guidance: for general-purpose use, full fine-tuning is still preferable, but for coding tasks, a smaller domain-specific model can be more efficient. The main limitation is the drop in multilingual performance after DPO, which the authors attribute to data imbalance—a clear call for future work to include multilingual preference data. Overall, Tulu 2 advances the open-source ecosystem by providing reproducible, high-performing models and insights that help practitioners make informed trade-offs.