LlamaFactory

Free

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Model APIsFreeFree tier

Inputs: textOutputs: text

Type

Open Source

Visit Website GitHub (70.9k stars)

About LlamaFactory

LLaMA Factory is a user-friendly and efficient platform for training and fine-tuning large language models (LLMs) and vision-language models (VLMs). It supports over 100 pre-trained models including LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen, Yi, Gemma, Baichuan, ChatGLM, and Phi. The platform offers a wide array of training algorithms such as incremental pre-training, multimodal instruction supervised fine-tuning, reward model training, PPO, DPO, KTO, and ORPO. It enables flexible precision tuning with 16-bit full, frozen, LoRA, and QLoRA (2/3/4/5/6/8 bits) via AQLM/AWQ/GPTQ/HQQ/EETQ. Optimization algorithms like GaLore, BAdam, DoRA, LongLoRA, LLaMA Pro, LoRA+, LoftQ, and PiSSA are integrated, along with acceleration operators (FlashAttention-2, Unsloth) and inference engines (Transformers, vLLM). Experiment monitoring includes LlamaBoard, TensorBoard, Wandb, MLflow, and SwanLab. Users can run training without writing code through a WebUI and leverage distributed training with NativeDDP, DeepSpeed, FSDP, FSDP2, and Ray.

Key Features

Supports 100+ pre-trained models (LLaMA, LLaVA, Mistral, Qwen, Yi, ChatGLM, etc.)

Multiple training algorithms: pre-training, SFT, reward model, PPO, DPO, KTO, ORPO

Precision flexibility: full 16-bit, frozen, LoRA, QLoRA (2–8 bit) with AQLM/AWQ/GPTQ/HQQ

Optimization algorithms: GaLore, BAdam, DoRA, LoRA+, LoftQ, PiSSA, LongLoRA, LLaMA Pro

Acceleration operators: FlashAttention-2, Unsloth, Liger Kernel

Inference engines: Transformers and vLLM

Experiment monitoring: LlamaBoard, TensorBoard, Wandb, MLflow, SwanLab

No-code WebUI for training without programming

Distributed training: NativeDDP, DeepSpeed, FSDP, FSDP2, Ray

Pros & Cons

Pros

Broad model and algorithm support (100+ models, multiple training methods)
Flexible precision options from full to 2-bit QLoRA accommodate diverse hardware
No-code WebUI lowers entry barrier for non-programmers
Integrated experiment monitoring across multiple platforms
Open-source and free with active development (ACL 2024 publication)
Distributed training support scales to multi-node setups

Cons

Documentation primarily in Chinese, which may be a barrier for some users
Advanced configuration (e.g., custom optimization, multi-modal data) requires understanding of training concepts
Training large models still demands significant GPU resources despite quantization options

Best For

Fine-tuning LLMs for domain-specific tasks (e.g., legal, medical, financial)Multimodal instruction fine-tuning with vision-language models (e.g., LLaVA)Reinforcement learning from human feedback (RLHF) for alignmentEfficient fine-tuning with limited GPU memory using LoRA/QLoRAModel evaluation and comparison of fine-tuned checkpointsExporting quantized models for deployment

Alternatives to LlamaFactory

AutoGPT

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

aider

aider is AI pair programming in your terminal

axolotl

Go ahead and axolotl questions

Chroma

Open-source embedding database

awesome-claude-code

A curated list of awesome skills, hooks, slash-commands, agent orchestrators, applications, and plugins for Claude Code by Anthropic

agents-course

This repository contains the Hugging Face Agents Course.

FAQ

What models does LLaMA Factory support?

It supports over 100 pre-trained LLMs and VLMs, including LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen, Yi, Gemma, Baichuan, ChatGLM, and Phi.

Can I use LLaMA Factory without writing code?

Yes, LLaMA Factory provides a no-code WebUI for training, allowing users to fine-tune models without programming.

Is LLaMA Factory free?

Yes, it is open-source and free to use for both research and commercial purposes.

How do I install LLaMA Factory?

The documentation includes installation instructions for standard environments, NPU support, and Docker images for NPU.