Self Gradient Forcing: Native Long Video Extrapolation

Analysis

Why This Paper Matters

Autoregressive video generation has been limited by exposure bias and the inability to generate long, coherent videos beyond the training context window. Self Forcing partially addresses exposure bias by training on self-generated histories, but it fails to provide supervision for how earlier latents should be encoded into the key-value cache for future frames. This paper identifies and solves the historical context-gradient gap, a fundamental limitation in current autoregressive video models. By enabling future losses to supervise past context encoding, SGF allows models to learn more effective causal memory, which is crucial for maintaining subject identity and temporal consistency over long horizons.

The ability to extrapolate to several-minute-long videos from a 5-second training window is remarkable and suggests that SGF may unlock practical applications in long-form video generation, such as movie production, virtual environments, and video game cutscenes. This work is particularly relevant as the field moves toward native autoregressive video generation without reliance on external modules or post-processing.

Technical Contributions

• Historical context-gradient gap: Identifies that in Self Forcing, the key-value cache from earlier frames is treated as frozen rollout state, preventing future losses from supervising how those latents are written into memory.
• Two-pass training strategy: Pass 1 performs a no-gradient rollout to record self-generated context and noisy latents at a sampled denoising exit step. Pass 2 recomputes context KV representations and future-to-context causal attention, allowing gradients from future losses to flow back to the context encoding.
• Stop-gradient clean-latent input: The self-generated context is used as stop-gradient input, ensuring that the model learns to encode better context without interfering with the rollout dynamics.
• Native autoregressive objective: SGF integrates memory-writing supervision directly into the training loss without architectural modifications or additional modules.

Results

The paper reports that SGF achieves stronger native long-video extrapolation than Self Forcing across multiple metrics: subject identity, background/layout consistency, and temporal stability. The experiments cover both frame-wise and chunk-wise generation under different initialization conditions. The most striking result is that with only a 5-second training window, SGF can extrapolate to videos lasting several minutes, demonstrating a dramatic improvement in temporal coherence. No specific numerical metrics (e.g., FID, FVD) are provided in the abstract, but the qualitative improvements are emphasized.

Significance

SGF addresses a core limitation in autoregressive video generation, enabling models to learn more effective causal memory without architectural changes. This could reduce the need for large training datasets and long context windows, making long-form video generation more accessible. The approach is general and could be applied to other autoregressive generative tasks, such as audio or text generation, where long-range dependencies are critical. By releasing code and models, the authors aim to accelerate research in this direction.