GPT-NeoX

GPT-NeoX is EleutherAI's open-source library for training large-scale autoregressive language models on GPUs. Built on NVIDIA's Megatron Language Model and augmented with techniques from DeepSpeed, it provides distributed training with ZeRO and 3D parallelism (data, tensor, and pipeline). The library supports a wide variety of hardware and cluster schedulers, including AWS, CoreWeave, ORNL Summit/Frontier, LUMI, and launching via Slurm, MPI, and IBM Job Step Manager. It implements cutting-edge architectural innovations such as rotary and alibi positional embeddings, parallel feedforward attention layers, and flash attention. GPT-NeoX comes with predefined configurations for popular architectures like Pythia, PaLM, Falcon, and LLaMA 1&2, and offers curriculum learning, integration with Hugging Face tokenizers/transformers, experiment monitoring via WandB, Comet, TensorBoard, and evaluation via the Language Model Evaluation Harness. Recent additions include Transformer Engine integration, preference learning (DPO, KTO, reward modeling), Mixture-of-Experts (MoE), and RWKV pipeline parallelism. It is used by researchers at Oak Ridge National Lab, Carnegie Mellon University, Stability AI, and others. Note: GPT-NeoX is designed for training billion‑parameter models from scratch; for generic inference, the Hugging Face transformers library is recommended.