Neura Market
ROLL logo

ROLL

Free

An Efficient and User-Friendly Scaling Library for Reinforcement Learning with Large Language Models.

FreeFree tier
Inputs: textOutputs: text
Type
Open Source
Company
Alibaba
Visit Website GitHub (38 stars)

About ROLL

ROLL is an efficient and user-friendly reinforcement learning library developed by Alibaba for large language models (LLMs), designed to run at scale on large GPU clusters. It leverages a multi-role distributed architecture built on Ray for flexible resource allocation and heterogeneous task scheduling, and integrates advanced inference and training frameworks such as Megatron-Core, SGLang, and vLLM. ROLL specializes in human preference alignment (RLHF), complex reasoning tasks (e.g., mathematics, code), and multi-turn agentic interaction scenarios. It supports a wide range of model architectures including dense and mixture-of-experts (MoE) models, Qwen3.5, Qwen3-Omni, and offers training strategies like FSDP2, Megatron with LoRA, partial GPU overlapping, on-policy distillation, and tool-use training. Recent updates also add support for Ascend NPUs and video reward models (Wan2_2 Reward FL). The library is open source and available on GitHub under the Alibaba organization.

Key Features

Multi-role distributed architecture with Ray for flexible resource allocation and heterogeneous task scheduling
Integration with Megatron-Core, SGLang, and vLLM for accelerated training and inference
Supports human preference alignment (RLHF), complex reasoning, and multi-turn agentic interaction
Training strategies: FSDP2, Megatron with LoRA, GPU partial overlapping, on-policy distillation
Support for dense and MoE models, Qwen3.5, Qwen3-Omni, and Ascend NPU
Tool use training via GEM environment definition and agentic capabilities
Video reward model support (Wan2_2 Reward FL)
Active development with frequent updates and paper releases

Pros & Cons

Pros
  • Efficient scaling with large GPU clusters and distributed architecture
  • User-friendly design with modular components and clear documentation
  • Integrates cutting-edge inference and training technologies (Megatron-Core, vLLM, SGLang)
  • Active community and frequent releases with new features and supported models
  • Open source and free to use with permissive license
Cons
  • Requires significant GPU resources for large-scale training
  • Dependency on multiple external systems (Ray, Megatron, vLLM) increases setup complexity
  • Documentation may not cover all advanced features in depth due to rapid development

Best For

Human preference alignment (RLHF) for LLMsComplex reasoning tasks such as mathematics, code generation, and logical inferenceMulti-turn agentic interaction and tool use trainingOn-policy distillation from teacher modelsVideo reward modeling and multimodal RL training

FAQ

What is ROLL?
ROLL is an open-source reinforcement learning library by Alibaba, designed to scale RL training for large language models using multi-role distributed architectures with Ray, and integrating Megatron-Core, SGLang, and vLLM.
What models and architectures does ROLL support?
ROLL supports dense and mixture-of-experts (MoE) models, including the Qwen3.5 series, Qwen3-Omni, and others. It also supports Ascend NPU and video reward models (Wan2_2 Reward FL).
What training strategies are available in ROLL?
ROLL offers FSDP2, Megatron with LoRA, GPU partial overlapping, on-policy distillation, and tool use training via GEM environment definitions.
Is ROLL free and open source?
Yes, ROLL is open source and hosted on GitHub under the Alibaba organization. It is free to use and modify.