Mistral Inference

Mistral Inference is the official Python inference library developed by Mistral AI for running Mistral's open-weight models locally on user hardware. It provides a minimal, high-performance codebase for downloading, loading, and performing inference with the full Mistral model family, including Mistral 7B, Mixtral 8x7B, Mixtral 8x22B, Codestral 22B, Codestral Mamba, Mathstral, Mistral Nemo, Mistral Large 2, Pixtral 12B, and Mistral Small 3.1. The library offers both a command-line interface for quick testing and a Python API for programmatic inference, with support for multi-GPU setups via torchrun.

All supported models include function calling capabilities, and certain models are available under custom licenses for research and non-commercial use. The library is designed for flexibility, allowing users to run models locally for tasks such as text generation, code completion, mathematical reasoning, and multimodal understanding (for Pixtral models). It is distributed as an open-source package on PyPI and can also be installed from source.

Notably, the official Mistral Inference repository on GitHub has been archived and is now read-only, as indicated by the repository status. Users should verify the current maintenance status and consider alternatives for latest updates. The library requires a GPU for installation due to its dependency on xformers, and inference may require significant hardware resources for larger models.