Neura Market
MInference logo

MInference

Free

To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

FreeFree tier
Inputs: textOutputs: text
Type
Open Source
Company
Microsoft
Visit Website GitHub (38 stars)

About MInference

MInference is an open-source library by Microsoft for accelerating long-context LLM inference. It leverages the dynamic sparse nature of LLMs' attention, which exhibits static patterns per head, to speed up the pre-filling stage. The method first determines offline which sparse pattern each attention head belongs to, then approximates the sparse index online and dynamically computes attention using optimized custom kernels. This approach achieves up to a 10x speedup for pre-filling on an A100 GPU while maintaining accuracy. It supports million-token prompts and works with long-context LLMs such as LLaMA-3-8B-1M and GLM-4-1M. MInference has been integrated into frameworks like SGLang and vLLM. The project also includes SCBench for KV cache-centric evaluation and MMInference for multi-modal long-context VLMs. MInference was accepted as a NeurIPS'24 Spotlight, ICLR'25, and ICML'25.

Key Features

Dynamic sparse attention leveraging static patterns per head
Offline sparse pattern determination per attention head
Online sparse index approximation and dynamic computation
Optimized custom kernels for efficient attention calculation
Up to 10x speedup for pre-filling on A100 GPUs
Supports 1M-token context with models like LLaMA-3-8B-1M and GLM-4-1M
Integration with SGLang and vLLM inference frameworks
Includes SCBench for KV cache evaluation and MMInference for multi-modal VLMs
Accepted at NeurIPS'24 (Spotlight), ICLR'25, and ICML'25

Pros & Cons

Pros
  • Up to 10x speedup on A100 while preserving accuracy
  • Open source with clear documentation and examples
  • Integrates with popular LLM serving frameworks (vLLM, SGLang)
  • Validated by acceptance at top ML conferences (NeurIPS, ICLR, ICML)
  • Supports very long contexts (million tokens) on a single GPU
Cons
  • Speedup primarily targets pre-filling stage; decode stage benefit may vary
  • Requires offline profiling to determine sparse patterns for each model
  • Optimized for specific GPU architectures (e.g., A100)

Best For

Accelerating pre-filling stage of long-context LLM inferenceHandling million-token prompts for tasks like document analysis and long-form generationReducing latency in applications relying on transformer-based language modelsResearch and deployment of efficient long-context attention mechanisms

FAQ

What is MInference?
MInference is an open-source method and library from Microsoft that speeds up long-context LLM inference by applying approximate dynamic sparse attention, achieving up to 10x pre-filling speedup on an A100.
Which models does MInference support?
MInference supports long-context LLMs such as LLaMA-3-8B-1M and GLM-4-1M, and has been integrated into Qwen2.5-1M and Meta-Llama-3.1-8B-Instruct.
How much speedup can I expect?
MInference provides up to 10x speedup for the pre-filling stage on an A100 GPU, with speedup factors scaling with context length (e.g., 5.2x at 256K, 8x at 512K, 15x at 1M when using SGLang optimizations).
Is MInference free and open source?
Yes, MInference is open source under a permissive license and free to use. The code is available on GitHub at https://github.com/microsoft/MInference.