A comprehensive survey on pretrained foundation models: A history from bert to chatgpt

Analysis

Why This Paper Matters

This survey arrives at a critical juncture where pretrained foundation models (PFMs) have become the backbone of modern AI. By tracing the lineage from BERT (2018) to ChatGPT and GPT-4 (2023-2024), the paper offers a structured narrative that helps practitioners understand why certain architectural choices (e.g., encoder-only vs. decoder-only) and training paradigms (e.g., masked language modeling vs. autoregressive next-token prediction) succeeded. For Neura Market's audience of AI practitioners, this is invaluable: it distills the key design decisions that led to the current dominance of large language models and highlights the emerging trend of multi-modal PFMs that unify text, images, and code.

The paper also addresses the elephant in the room—scaling laws and emergent abilities. It documents how increasing model size, data volume, and compute leads to qualitatively new capabilities (e.g., in-context learning, chain-of-thought reasoning) that were not explicitly programmed. This is directly relevant for practitioners deciding whether to train from scratch, fine-tune, or prompt-engineer existing PFMs.

Technical Contributions

• Taxonomy of PFMs: Organizes models into three generations: (1) encoder-only (BERT, RoBERTa), (2) encoder-decoder (T5, BART), and (3) decoder-only (GPT series, LLaMA). This clarifies the trade-offs between bidirectional context and generative flexibility.
• Training Paradigms: Covers self-supervised pre-training (MLM, autoregressive), instruction tuning (FLAN, InstructGPT), and alignment techniques (RLHF, DPO). The survey explains how each stage contributes to model utility and safety.
• Multi-Modal Extensions: Reviews models like CLIP (text+image), Flamingo (few-shot vision-language), and GPT-4V (vision-language). It shows how PFMs are increasingly becoming universal interfaces across modalities.
• Scaling and Emergence: Summarizes empirical findings from Kaplan et al. (2020) and Hoffmann et al. (2022) on optimal scaling, and discusses emergent abilities observed at scale (e.g., arithmetic, translation).

Results

The survey does not present new experimental results but synthesizes reported metrics from the literature. Key quantitative takeaways include: BERT achieved GLUE score 80.5 (2018), GPT-3 reached 175B parameters with few-shot performance rivaling fine-tuned models, and ChatGPT (GPT-3.5) improved instruction following by 10-20% over GPT-3 on human evaluation. GPT-4 reportedly scores in the top 10% on the Uniform Bar Exam and achieves near-human performance on various professional exams. The paper also notes that scaling data quality (e.g., deduplication, filtering) is as important as scaling model size.

Significance

This survey provides a unified framework for understanding the rapid evolution of PFMs, which is essential for AI practitioners navigating a fragmented landscape. It highlights that the field is moving toward larger, more aligned, and multi-modal models, with safety and efficiency becoming paramount. For Neura Market readers, the paper's main value is as a roadmap: it identifies which architectural innovations (e.g., mixture of experts, sparse attention) and training techniques (e.g., RLHF, constitutional AI) are likely to shape the next generation of PFMs. The survey also underscores the importance of open-source models (LLaMA, Mistral) in democratizing access, a trend with significant implications for deployment and customization.