Neura Market
lorax logo

lorax

Free

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Model APIsFreeFree tier
Type
Open Source
Company
Predibase
Visit Website GitHub (3.8k stars)

About lorax

LoRAX (LoRA eXchange) is an open-source framework for serving thousands of fine-tuned LLMs on a single GPU, dramatically reducing cost without compromising throughput or latency. It supports dynamic loading of LoRA adapters from HuggingFace, Predibase, or any filesystem, and merges adapters per request for instant ensembles. Features include heterogeneous continuous batching, adapter exchange scheduling, optimized inference with tensor parallelism, flash-attention, paged attention, quantization, and token streaming. Production-ready with Docker images, Kubernetes Helm charts, Prometheus metrics, OpenTelemetry tracing, and an OpenAI-compatible API supporting multi-turn chat and structured JSON output. Released under Apache 2.0 license for free commercial use.

Key Features

Dynamic adapter loading from HuggingFace, Predibase, or any filesystem without blocking concurrent requests
Merge adapters per request to create instant ensembles
Heterogeneous continuous batching packs different adapters together, keeping latency nearly constant
Adapter exchange scheduling prefetches and offloads adapters between GPU and CPU memory
Optimized inference with tensor parallelism, flash-attention, paged attention, SGMV kernels, quantization, token streaming
Production-ready: Docker images, Helm charts, Prometheus metrics, OpenTelemetry tracing, OpenAI-compatible API
Private adapters with per-request tenant isolation
Structured output (JSON mode)
Free commercial use under Apache 2.0 license

Pros & Cons

Pros
  • Free for commercial use (Apache 2.0 license)
  • Dramatically reduces cost by serving many fine-tuned models on one GPU
  • High throughput and low latency with advanced batching and optimization
  • Supports a wide range of base models (Llama, Mistral, Qwen, etc.) and quantization methods (bitsandbytes, GPT-Q, AWQ)
  • Dynamic adapter loading without blocking – adapters loaded just-in-time
  • Production-ready with Docker, Kubernetes, monitoring, and tracing
  • OpenAI-compatible API for easy integration
Cons
  • Requires Nvidia GPU Ampere generation or higher
  • Requires CUDA 11.8+ and Linux OS
  • Setup complexity – recommended to use pre-built Docker images and may need custom CUDA kernel compilation
  • Only supports LoRA adapters (not full fine-tuning or other adapter types)
  • Primarily designed for text generation; multimodal support not mentioned

Best For

Serve thousands of fine-tuned LLMs on a single GPU to drastically reduce serving costsDynamically load and swap LoRA adapters for task-specific inference without redeploymentCombine multiple adapters into ensembles for improved model performanceDeploy multi-turn chat applications with an OpenAI-compatible APIProduce structured JSON outputs from LLMs for downstream automation

Alternatives to lorax

AutoGPT

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

aider

aider is AI pair programming in your terminal

axolotl

Go ahead and axolotl questions

Chroma

Open-source embedding database

awesome-claude-code

A curated list of awesome skills, hooks, slash-commands, agent orchestrators, applications, and plugins for Claude Code by Anthropic

agents-course

This repository contains the Hugging Face Agents Course.

FAQ

What is LoRAX?
LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.
What base models are supported?
LoRAX supports Llama (including CodeLlama), Mistral (including Zephyr), and Qwen as base models. Base models can be loaded in fp16 or quantized with bitsandbytes, GPT-Q, or AWQ. See the documentation for a complete list of supported architectures.
How do I get started with LoRAX?
The recommended way is to use the pre-built Docker image. Run 'docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/predibase/lorax:main --model-id <model>' and then prompt via REST API or Python client. A full tutorial is available in the Getting Started guide.