DeepSpeed ZeRO++

DeepSpeed ZeRO++ is a system optimization from Microsoft Research that dramatically reduces communication overhead during large-scale deep learning training. Building on the ZeRO (Zero Redundancy Optimizer) family, ZeRO++ achieves up to a 4x reduction in communication volume compared to ZeRO stage 3, enabling faster training of large language models (LLMs) and chat models (e.g., Turing-NLG, Bloom, ChatGPT) even in bandwidth-constrained or small per-GPU batch size scenarios. It employs techniques such as quantized weights and gradients, data remapping, and communication remapping to minimize data transfer. ZeRO++ integrates seamlessly with the DeepSpeed framework and DeepSpeed-Chat, offering up to 28-36% throughput improvement in high-bandwidth clusters and up to 2.2x speedup in low-bandwidth environments. It also enhances Reinforcement Learning from Human Feedback (RLHF) training, achieving 1.3x speedup in training and 2x speedup in token generation for dialogue models.