OpenScholar

Analysis

Why This Paper Matters

The explosion of scientific publications makes it nearly impossible for researchers to stay current. OpenScholar addresses this by creating a specialized retrieval-augmented language model that can answer complex scientific queries by synthesizing information from 45 million open-access papers, complete with accurate citations. This is a significant step toward automating literature review while maintaining the transparency and verifiability essential in scientific work.

How It Works

OpenScholar operates through a three-component pipeline. First, a datastore D contains 234 million passages derived from 45 million open-access papers from the peS2o v3 corpus, each passage being a 250-word block with the paper title attached. Second, a retrieval pipeline R uses a bi-encoder retriever (continually pre-trained Contriever) to fetch initial passages from three sources: the peS2o datastore, Semantic Scholar API for abstracts, and web search via You.com restricted to academic platforms. A cross-encoder reranker (fine-tuned BGE-reranker with synthetic data from Llama 3 70B) then refines the top passages, applying meta-filters like limiting passages per paper and incorporating normalized citation counts.

The generator LM G is an 8B model (Llama 3.1 8B Instruct) trained on synthetic data. Training data is generated by sampling 1 million paper abstracts, selecting 10,000 papers published after 2017, and using Llama 3.1 70B to create literature review questions and answers via the OpenScholar pipeline. A two-step filtering process (pairwise and rubric) ensures quality. The model is trained on three types of data: answer generation, feedback generation, and feedback incorporation, blended with general-domain and scientific instruction-tuning data (50% each).

During inference, the model uses iterative generation with self-feedback. It first produces an initial response y0 with citations, then generates up to three feedback sentences to improve the output. It can also issue new retrieval queries. This process repeats until all feedback is addressed. Finally, citation verification ensures every citation-worthy statement has a supporting reference, with post hoc insertion if needed.

Results

The authors introduce ScholarQABench, a comprehensive benchmark for evaluating literature search. It includes single-paper tasks (SciFact, PubMedQA, QASA) and multi-paper long-form QA datasets across computer science (ScholarQA-CS, 100 questions), biomedicine and neuroscience (ScholarQA-BIO and ScholarQA-NEURO, 2,759 questions), and four domains (ScholarQA-MULTI, 108 questions). All annotations are done by domain experts with Ph.D.s.

Evaluation uses a multifaceted automatic pipeline: correctness (accuracy, ROUGE-L, or rubric-based scoring), citation accuracy (precision, recall, F1), and content quality (relevance, coverage, organization via Prometheus v2). The model shows strong performance, particularly in multi-paper synthesis tasks, though the authors note that initial drafts (y0) are sometimes preferred over refined versions (yT) about 20% of the time due to over-editing.

Significance

OpenScholar represents a major advance in making scientific literature accessible and synthesizable. By open-sourcing all components—datastore, retrievers, model, and benchmark—it enables reproducibility and further development. The iterative self-feedback mechanism with citation verification addresses a critical need for trustworthy AI-generated scientific content. However, reliance on synthetic data and potential over-editing are limitations. The benchmark itself is a valuable resource for future research. Overall, OpenScholar could significantly reduce the time researchers spend on literature review, accelerating scientific discovery.