Dolma

Analysis

Why This Paper Matters

Large language models (LLMs) are increasingly central to AI, but their training data often remains proprietary or poorly documented. This lack of transparency hinders reproducibility and scientific progress. Dolma addresses this gap by releasing a 3 trillion token open corpus, along with a curation toolkit, enabling researchers to study the effects of data composition, quality, and scale on model behavior. The paper systematically documents design choices—from filtering to deduplication—and provides empirical evidence on how these choices affect downstream performance. For practitioners, Dolma offers a standardized foundation for pretraining experiments, reducing the guesswork in assembling large datasets.

How It Works

Dolma is assembled from seven sources: Common Crawl (web), The Stack (code), Pushshift Reddit (conversational forums), C4 (curated web), PeS2o (academic papers), Project Gutenberg (books), and Wikipedia/Wikibooks (encyclopedic content). The curation pipeline applies four transformations:

• Language filtering: Uses fastText's language ID model to retain English text. The authors acknowledge that this is imperfect, leaving some multilingual content.
• Quality filtering: Reimplements heuristics from C4 and Gopher (e.g., removing documents with too few words or too many punctuation marks) rather than using model-based quality filters.
• Content filtering: Combines rule-based and classifier-based toxicity filtering, including a fastText tagger for NSFW and hate speech detection.
• Deduplication: Applies URL, document, and paragraph-level deduplication using Bloom filters for linear-time efficiency. Deduplication is performed within each source but not across sources.

For the web subset, data is drawn from Common Crawl dumps between 2020–2022. The most common domains by token count are patents.google.com, nature.com, and frontiersin.org. 63.6% of web documents come from .com sites. The code subset comes from The Stack, and the conversational subset from Pushshift Reddit.

Results

The paper evaluates decontamination and mixing strategies using 1B parameter models trained on subsets of Dolma. Key findings:

• Decontamination: Removing documents overlapping with the Paloma benchmark removed less than 0.001% of characters and fewer than 0.02% of documents. Perplexity and downstream task performance showed minimal and inconsistent differences, indicating no significant degradation.

• Code mixing: Adding code (5% or 15%) to pretraining improved in-context learning on bAbI and WebNLG tasks. For GSM8k, models pretrained with code outperformed the C4-only model when fine-tuned on program-aided outputs.

• Subset mixing: Different mixtures (e.g., Web-only, Reference+, Gopher-like) showed similar perplexity on web documents but varied on specialized domains. Models trained with even small amounts of in-domain data (e.g., academic papers) achieved lower perplexity on those domains.

Significance

Dolma fills a critical gap in open LLM research by providing a large, well-documented corpus that enables fair comparisons across models and training recipes. The accompanying toolkit lowers the barrier for researchers to replicate and extend data curation pipelines. The empirical findings on decontamination and mixing offer practical guidance for practitioners: decontamination has negligible impact on performance, and including code benefits reasoning tasks. However, the English-only scope and imperfect language filtering limit its applicability to multilingual settings. Overall, Dolma sets a new standard for openness and reproducibility in pretraining data, likely influencing future dataset releases and model development.