Preprint2024
Instruction Pretraining
Unknown
Instruction Pre-Training augments raw corpora with instruction-response pairs for supervised multitask pretraining of language models.
0Jun 1, 2024Instruction Tuning
arXiv
AI Research
AI & ML Research Papers
A comprehensive index of artificial intelligence and machine-learning research with AI-generated summaries, citation metrics, and direct links to papers and code.
All Papers
14 papersPreprint2024
Fine Web
Unknown
Fine Web is a 15 trillion token dataset for pretraining LLMs that yields better-performing models than other open datasets.
0May 1, 2024
Preprint2024
Dolma
Unknown
Dolma is an open corpus of three trillion tokens for language model pretraining, designed to support transparency and reproducibility in AI research.
0Jan 1, 2024
arXiv
Preprint2022
Matcha
Unknown
Introduces pretraining tasks on math reasoning and chart derendering to enhance chart comprehension in diverse visual language tasks.
0Dec 1, 2022Reasoning
Preprint2024
Autoregressive Image Models (AIM)
Unknown
AIM shows that autoregressive pretraining on images scales effectively with model and data size, achieving strong performance on vision tasks.
0Jan 1, 2024Large Language ModelsAttention Mechanisms
PreprintComputer Vision and Pattern Recognition2022
Swin Transformer V2
Ze Liu, Han Hu, Yutong Lin, et al.
Swin Transformer V2 introduces post-norm, cosine attention, log-spaced position bias, and self-supervised pretraining to scale vision transformers to 3B parameters.
2.9kApr 1, 2022Transformers
arXiv
Preprint2025
SigLIP 2
Unknown
SigLIP 2 improves multilingual vision-language encoders with captioning pretraining, self-supervised losses, and online data curation, offering native aspect ratio preservation.
0Feb 1, 2025Computer Vision
Preprint2020
SimCLRv2
Unknown
SimCLRv2 presents a semi-supervised learning framework combining unsupervised pretraining, supervised fine-tuning, and distillation with unlabeled data.
0Jun 1, 2020Fine Tuning
Preprint2025
UltraLong
Unknown
Introduces a training method for developing ultra-long context LLMs with context windows extending up to 4 million tokens via efficient continued pretraining with YaRN-based scaling and instruction tuning.
0Apr 1, 2025Scaling LawsInstruction Tuning
PreprintInternational Conference on Learning Representations2023
Llemma
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, et al.
Llemma is an open-source LLM for mathematics, built by continued pretraining of Code Llama on a curated dataset of scientific papers, web math, and code, enabling tool use and formal theorem proving.
479Oct 1, 2023
arXiv
Preprint2023
LLaMA 2 Long
Unknown
LLaMA 2 Long extends LLaMA 2 to handle up to 32,768 tokens via continual pretraining with modified RoPE and synthetic instruction data.
0Sep 1, 2023
arXiv
PreprintNeural Information Processing Systems2019Featured
XLNet
Zhilin Yang, Zihang Dai, Yiming Yang, et al.
XLNet combines autoregressive and autoencoding pretraining via permutation language modeling, outperforming BERT on NLP tasks.
9.5kJun 1, 2019Transformers
arXiv