A flexible, extensible AI agent backend built with NestJS—designed for running local, open-source LLMs (Llama, Gemma, Qwen, DeepSeek, etc.) via Docker Model Runner. Real-time streaming, Redis messaging, web search, and Postgres memory out of the box. No cloud APIs required!

Local Agent: Fast AI Backend with Docker Model Runner

A flexible, extensible AI agent backend built with NestJS—designed for running local, open-source LLMs (Llama, Gemma, Qwen, DeepSeek, etc.) via Docker Model Runner. Real-time streaming, Redis messaging, web search, and Postgres memory out of the box. No cloud APIs required!

🚀 Quick Start

1. Clone the repository

   git clone <your-repo-url>
   cd <your-repo-folder>
   

2. Copy and edit environment variables

   cp .env.example .env
   # Edit .env and fill in your model and service config
   

3. Start required services (Redis, PostgreSQL, Local LLM) with Docker Compose

   docker compose up -d
   

   • PostgreSQL: localhost:5433
   • Redis: localhost:6379
   • Local LLM runner: localhost:12434 (Model Runner guide)
4. Install dependencies

   pnpm install
   

5. Start the development server

   pnpm run start:dev
   

🛠️ Environment Variables

See .env.example for all options. Key variables:

• MODEL_BASE_URL — e.g. http://localhost:12434/engines/llama.cpp/v1
• MODEL_NAME — e.g. ai/gemma3:latest, llama-3, qwen, deepseek
• TAVILY_API_KEY — for web search (Get your key)
• REDIS_HOST, REDIS_PORT, etc.
• POSTGRES_* — for memory

✨ Features

• 🤖 Local, open-source LLMs (Llama, Gemma, Qwen, DeepSeek, etc.)
• 🌊 Real-time streaming responses
• 💾 Conversation history with Postgres memory
• 🌐 Web search integration (Tavily)
• 🧵 Custom ThreadService for conversations
• 📡 Redis pub/sub for real-time messaging
• 🎯 Clean, maintainable architecture

🧩 Model Setup (Docker Model Runner)

• This project is designed for local LLMs only, using Docker Model Runner.
• Supported models: Llama, Gemma, Qwen, DeepSeek, and other op