Instruction Pretraining

Analysis

Why This Paper Matters

Traditional language model pretraining treats raw text as a monolithic stream, learning general patterns but lacking explicit task awareness. Instruction Pre-Training flips this by embedding task-specific supervision directly into the pretraining data. This matters because it bridges the gap between unsupervised pretraining and supervised fine-tuning, making models more capable from the start. The framework is especially valuable for domain adaptation, where labeled data is scarce, and for improving data efficiency—achieving strong performance with fewer tokens.

How It Works

The core idea is simple: instead of pretraining on raw text alone, each document is augmented with a set of instruction-response pairs generated by an instruction synthesizer. This turns pretraining into a supervised multitask learning process.

Instruction Synthesizer: The synthesizer is built by fine-tuning Mistral-7B on a diverse collection of datasets. For each context (raw text), all associated downstream tasks are gathered as instruction-response pairs. To ensure diversity, at most 10K examples per dataset are sampled. The synthesizer is trained to generate both one-shot and few-shot examples. During inference, it performs multi-round generation, producing about 5 pairs per raw text, each averaging 52 tokens.

LM Pre-Training: The pretraining setup remains standard—next-token prediction with loss on all tokens. For general pretraining from scratch, only 1/5 of the raw corpora (40M texts out of 200M) is converted to instruction-augmented form, leaving the rest unchanged to manage data volume. For domain-adaptive continual pretraining, all raw corpora are augmented and mixed with general instructions to retain prompting ability.

Results

General Pretraining: Instruction Pre-Training outperforms both vanilla pretraining and a mixed baseline that simply adds fine-tuning data. The 1.3B parameter model matches or exceeds larger open-source models like Pythia-1B and BLOOM-3B while using fewer tokens. During instruction tuning, the instruction-pretrained model quickly surpasses the vanilla one and maintains a stable improvement trend.

Domain-Adaptive Continual Pretraining: On biomedicine (PubMed) and finance (financial news), Instruction Pre-Training consistently beats vanilla pretraining on domain-specific tasks. The 8B parameter Llama3 model, after instruction-augmented continual pretraining, achieves parity with or surpasses Llama3-70B on several benchmarks. The finance NER benchmark showed inconsistency, possibly due to data quality issues.

Significance

Instruction Pre-Training offers a practical recipe for injecting task supervision into pretraining without requiring manual annotation. It is particularly impactful for domain adaptation, where it can elevate a smaller model to compete with much larger ones. The approach is data-efficient and scalable, as demonstrated by the 200M synthesized pairs. Future work could explore better synthesizer designs, more diverse task categories, and integration with reinforcement learning from human feedback. The released models and data provide a strong foundation for the community.