FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation

Analysis

Why This Paper Matters

Video Diffusion Transformers have become the dominant architecture for high-resolution video generation, but their self-attention over long spatio-temporal sequences creates a severe computational bottleneck. While training-free sparse attention methods like Top-p routing reduce this cost, they introduce uneven per-head workloads under multi-GPU sequence parallelism, turning sparse attention into a rank-level straggler problem. FVAttn directly tackles this overlooked issue, making it highly relevant for practitioners deploying video generation models at scale.

The paper's focus on runtime load balancing rather than static partitioning is a practical advance. By migrating heavy heads via P2P communication and filling slack with additional computation, FVAttn achieves near-ideal load balance without retraining. This is particularly important for production systems where model weights are fixed and inference latency is critical.

Technical Contributions

• Runtime Load Balancing: Migrates a small number of heavy heads across GPUs using P2P communication to shorten the current critical path, reducing average load imbalance from 1.34 to 1.08.
• Slack-Aware Sparse Augmentation: Fills residual non-critical-rank slack with additional high-value blocks, improving GPU utilization without increasing latency.
• Overlap Scheduling: Hides migration and scheduling overhead behind existing computation, ensuring no net increase in inference time.
• Sparse-Routing Frontend: Combines Top-p routing with a Top-k safety floor and video-aware block organization to produce high-quality sparse masks.
• Materialized Mask Repair: Repairs the mask at runtime to ensure correctness after load balancing.

Results

On the step-distilled Wan2.2 I2V model, FVAttn delivers:

• 4.41x attention speedup over FlashAttention (baseline dense attention).
• 2.02-2.11x DiT inference speedup (end-to-end).
• Load imbalance reduction from 1.34 to 1.08 (ideal is 1.0).
• Competitive video quality (no degradation reported).

These results demonstrate that runtime load balancing can effectively eliminate the straggler problem introduced by adaptive sparse attention, achieving near-linear scaling under sequence parallelism.

Significance

FVAttn addresses a practical distributed computing challenge that arises naturally when combining sparse attention with sequence parallelism. Its training-free nature means it can be applied to existing video generation models without modification. The techniques—runtime load balancing, slack-aware augmentation, and overlap scheduling—are general and could be adapted to other domains where adaptive sparsity creates workload imbalance. This work moves sparse attention from a purely algorithmic optimization to a systems-level solution, enabling more efficient deployment of large video diffusion models.