AI Models
Petals Lets Users Run Large AI Models at Home Like BitTorrent
Petals is a decentralized platform that allows users to run large language models such as Llama 3.1, Mixtral, Falcon, and BLOOM on consumer-grade hardware by sharing computational resources in a peer-to-peer network. Users load only a portion of a model and join a network of others serving the remaining parts, enabling inference speeds of up to 6 tokens per second for Llama 2 70B and 4 tokens per second for Falcon 180B. The platform supports fine-tuning, custom sampling methods, and access to hidden states, combining the convenience of an API with the flexibility of PyTorch and Hugging Face Transformers.
Jul 232 minNeura News
