Airllm

Free

AirLLM 70B inference with single 4GB GPU

FreeFree tier

Inputs: textOutputs: text

Type

Open Source

Visit Website GitHub (18.0k stars)

About Airllm

AirLLM is an open-source library that dramatically reduces inference memory usage, enabling large language models (LLMs) with 70B parameters to run on a single 4GB GPU without quantization, distillation, or pruning. It supports even larger models like Llama 3.1 405B on 8GB and DeepSeek-V3 (671B) on approximately 12GB. AirLLM provides an easy-to-use AutoModel API that automatically detects model type, and it is compatible with a wide range of popular models including Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma, ChatGLM, QWen, Baichuan, Mistral, and InternLM. Additional features include model compression for up to 3x speedup, prefetching for 10% speed improvement, support for safetensors, CPU inference, and 8-bit/4-bit quantization options. The library runs on Python via pip and is designed for local inference on resource-constrained hardware.

Key Features

Runs 70B LLMs on a single 4GB GPU without quantization, distillation, or pruning

Supports models up to 671B parameters (DeepSeek-V3) on ~12GB GPU memory

AutoModel API automatically detects model type and simplifies inference

Model compression achieves up to 3x runtime speedup

Prefetching overlaps model loading and compute for 10% speed improvement

Supports 8-bit and 4-bit quantization for further memory reduction

Compatible with dozens of popular models (Qwen, Llama, DeepSeek, Phi, Gemma, ChatGLM, etc.)

Inference on CPU and MacOS supported

Open source with Apache-2.0 license

Pros & Cons

Pros

Dramatically reduces memory requirements, enabling large models on small GPUs
No need for quantization, distillation, or pruning to achieve memory savings
Wide model support through a unified AutoModel interface
Actively maintained with frequent updates and new model support
Free and open source with permissive Apache-2.0 license

Cons

Inference speed is slower than full GPU inference due to layer-by-layer loading
Still requires a minimum of 4GB GPU memory for 70B models; larger models need more
Not optimized for real-time or high-throughput production scenarios

Best For

Running large language models on consumer-grade GPUs with limited VRAM (4GB-12GB)Local LLM inference without cloud dependencies or costly hardwareEvaluating and experimenting with 70B+ parameter models on budget setupsDeploying LLMs in edge or low-resource environments