GitHub
FreeMultimodal AI operating your computer
FreeFree tier
About GitHub
The Self-Operating Computer Framework, released in November 2023 by OthersideAI, is an open-source framework that enables multimodal AI models to operate a computer by viewing the screen and executing mouse and keyboard actions to achieve a given objective. It was one of the first implementations of full computer-use using multimodal models. The framework currently integrates with GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL, and LLaVA (via Ollama), with plans to support additional models. It can be installed via pip and run with a simple command, making it accessible for developers and researchers exploring AI-driven computer automation.
Key Features
Supports multiple multimodal models: GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL, LLaVA
Views screen and executes mouse and keyboard actions like a human operator
Easy installation via pip: pip install self-operating-computer
Simple command-line interface: run with 'operate'
Open source with active development and community contributions
Pros & Cons
Pros
- Open source and free to use (only requires API keys for selected models)
- Supports a wide range of leading multimodal models
- Easy to install and run with a single pip command
- One of the earliest full computer-use frameworks, with an active community
Cons
- Requires API keys for most multimodal models, which may involve costs
- Local model (LLaVA) requires Ollama setup and ~5GB storage
- Currently optimized for macOS; Linux support available, Windows limited
- May need screen recording and accessibility permissions on macOS
Best For
Automating repetitive computer tasks using natural language objectivesResearch and experimentation with multimodal AI agent capabilitiesPrototyping autonomous computer-use applicationsEducational demonstrations of AI-driven computer control
FAQ
How do I install the Self-Operating Computer Framework?
Install using pip: pip install self-operating-computer
How do I run the framework?
Use the command 'operate' in the terminal after installation.
How do I change my OpenAI API key?
Run 'vim .env' to edit the .env file and replace the old key.
Which multimodal models are supported?
Currently supported: GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL, and LLaVA (via Ollama).
What are the system requirements?
macOS is recommended. Linux is supported. Windows has limited support (Ollama in Preview). The terminal needs Screen Recording and Accessibility permissions on macOS.