A comprehensive index of artificial intelligence and machine-learning research with AI-generated summaries, citation metrics, and direct links to papers and code.
Zorik Gekhman, G. Yona, Roee Aharoni, et al.
This paper shows that supervised fine-tuning introducing new factual knowledge to LLMs is learned slowly and linearly increases hallucination, supporting that factual knowledge is acquired during pre-training.
Unknown
CLIMB automates optimal data mixture discovery for language model pre-training via clustering and iterative bootstrapping.
Unknown
Instruction Pre-Training augments raw corpora with instruction-response pairs for supervised multitask pretraining of language models.
Unknown
LayoutLMv2 integrates text, layout, and image in a single multi-modal Transformer pre-training framework with new cross-modal tasks.
Chuwei Luo, Changxu Cheng, Qi Zheng, et al.
GeoLayoutLM explicitly models geometric relations in pre-training to enhance feature representation for document layout analysis.
Unknown
OWL ViT enables open-vocabulary object detection by adapting Vision Transformers with CLIP-style pre-training and bipartite matching loss.
Unknown
Shows self-supervised pre-training can produce general visual features by scaling with curated diverse data, proposing a combined approach for larger models and datasets.
Unknown
A suite of bilingual text embedding models supporting up to 8192 tokens, trained via modified BERT pre-training and multi-task contrastive learning.
Unknown
SigLip introduces a pairwise sigmoid loss for language-image pre-training, enabling larger batch sizes and better performance at smaller batch sizes without global normalization.
Unknown
CLIP learns visual representations from 400M image-text pairs via contrastive pre-training, enabling zero-shot transfer to diverse tasks.
Unknown
MM1 introduces a family of multimodal LLMs up to 30B parameters, achieving SOTA pre-training metrics through careful data mixing and scaling.
Unknown
BLIP-2 introduces Q-Former, a lightweight transformer module that bridges frozen image encoders and frozen LLMs for efficient vision-language pre-training.