Deep Learning Workload Scheduling in GPU Datacenters: A Survey

Analysis

Why This Paper Matters

Deep learning workloads are increasingly deployed in large-scale GPU datacenters, where efficient scheduling is critical to minimize costs and maximize resource utilization. Traditional schedulers designed for big data or HPC workloads fail to account for the unique characteristics of DL training and inference, such as dynamic resource demands, iterative execution, and heterogeneous hardware. This survey fills a gap by providing a comprehensive, structured overview of schedulers specifically tailored for DL workloads, making it a valuable resource for system researchers and datacenter operators.

The paper systematically categorizes existing work by workload type (training vs. inference), scheduling objectives (e.g., makespan, throughput, fairness), and resource utilization strategies (e.g., co-location, preemption, elasticity). This taxonomy helps practitioners quickly identify relevant approaches and understand trade-offs. The inclusion of a curated GitHub repository with paper summaries and code links further enhances its utility as a living reference.

Technical Contributions

• Comprehensive taxonomy: The paper organizes schedulers along two primary axes: workload type (training vs. inference) and scheduling objective (e.g., makespan minimization, resource utilization, fairness).
• Training schedulers: Covers techniques such as gang scheduling, elastic training, and dynamic resource scaling, highlighting how they address iterative, long-running jobs with variable resource needs.
• Inference schedulers: Discusses latency-aware, throughput-oriented, and batching strategies for serving models with strict service-level objectives (SLOs).
• Resource utilization: Explores co-location of training and inference jobs, preemption, and multi-instance GPU (MIG) partitioning to improve overall cluster efficiency.
• Future directions: Identifies emerging DL workloads (e.g., large language models, graph neural networks), advanced decision-making (e.g., reinforcement learning, online learning), and hardware heterogeneity as key areas for future research.

Results

As a survey, the paper does not present new experimental results. Instead, it synthesizes findings from over 100 cited papers. Key takeaways include: (1) traditional schedulers achieve less than 50% GPU utilization for DL workloads in many cases; (2) specialized DL schedulers can improve utilization by 20-40% and reduce job completion times by 30-60% compared to general-purpose schedulers; (3) co-location of training and inference can increase overall throughput by 1.5-2x under appropriate isolation mechanisms.

Significance

This survey provides a foundational reference for the rapidly evolving field of DL workload scheduling. By systematically mapping the design space, it enables researchers to identify open problems and practitioners to select appropriate scheduling strategies for their datacenters. The paper also highlights the growing importance of scheduling for emerging workloads like large language models, which have unique resource and communication patterns. As GPU datacenters become the backbone of AI infrastructure, efficient scheduling will be a key differentiator in operational cost and performance.