Cogvideo: Large-scale pretraining for text-to-video generation via transformers

Analysis

Why This Paper Matters

CogVideo is a landmark paper in text-to-video generation, marking the first successful large-scale pretraining of a transformer model for this task. Before CogVideo, text-to-video generation was limited to narrow domains or short clips with simple motion, often using GANs or recurrent networks. By scaling up to a transformer architecture and pretraining on a massive dataset, CogVideo showed that general-domain text-to-video generation is achievable, setting a new standard for the field.

The paper's key insight is to leverage the success of text-to-image pretraining. Instead of training from scratch, CogVideo initializes from a pretrained text-to-image model (CogView2) and adapts it to video. This transfer learning approach drastically reduces the data and compute required for video pretraining, which is particularly important given the scarcity of high-quality text-video pairs. This strategy has become a common recipe in later video generation models, including diffusion-based ones.

Technical Contributions

• Multi-stage pretraining: CogVideo first pretrains on text-image pairs, then fine-tunes on text-video pairs, allowing efficient knowledge transfer.
• Temporal attention mechanism: The model extends the image transformer with a temporal attention layer to capture motion and consistency across frames.
• Frame interpolation: To generate longer videos, CogVideo uses a hierarchical generation approach, first generating keyframes and then interpolating intermediate frames.
• Large-scale dataset: The authors collected a large-scale text-video dataset, which is crucial for training a general-domain model.
• Transformer architecture: CogVideo uses a unified transformer that processes both text and video tokens, enabling joint modeling of language and visual content.

Results

The paper reports state-of-the-art results on the MSR-VTT and UCF-101 benchmarks, with significant improvements in both automatic metrics (FID, FVD) and human evaluations. For example, CogVideo achieved a FVD of 129.8 on MSR-VTT, outperforming prior GAN-based methods by a large margin. Human evaluators also preferred CogVideo's outputs over those of previous models in terms of semantic consistency and temporal coherence. The model was able to generate diverse and realistic videos from complex text prompts, such as "a dog running in the park" or "a woman playing guitar."

Significance

CogVideo's impact extends beyond text-to-video generation. It demonstrated that large-scale pretraining with transformers can be successfully applied to multi-modal generative tasks, bridging the gap between language and video. The transfer learning approach from image to video has become a standard technique, influencing subsequent models like Video Diffusion Models and Make-A-Video. Moreover, CogVideo's open-source release and the accompanying dataset have accelerated research in the field, enabling many follow-up works. The paper also highlights the importance of scaling up both model size and data, a trend that continues to dominate AI research. Overall, CogVideo is a foundational work that has shaped the trajectory of video generation research.