Puppeteer & Crawl4AI microservice for web automation, scraping, and AI processing with Bull queues

PuppetMaster 🤖

A powerful microservice for web automation, scraping, and data processing, integrating Puppeteer for browser control and Crawl4AI for advanced crawling and AI-powered extraction.

<img src="https://devin.ai/assets/askdeepwiki.png" alt="Ask https://DeepWiki.com" height="20"/>

Features

• Puppeteer Core:
  • 🌐 Headless browser automation with Puppeteer and Chromium
  • 🖱️ Standard browser interactions: navigate, click, type, scroll, select
  • 🖼️ Screenshot generation (full page or element)
  • 📄 PDF generation
  • ⚙️ Custom JavaScript evaluation
• Crawl4AI Integration:
  • 🕷️ Advanced crawling strategies (schema-based, LLM-driven)
  • 🧩 Flexible data extraction (CSS, XPath, LLM)
  • 🧠 Dynamic schema generation using LLMs
  • ✅ Content verification
  • 🔗 Deep link crawling
  • ⏳ Element waiting and filtering
  • 📄 PDF text extraction
  • 📝 Webpage to Markdown conversion
  • 🌐 Webpage to PDF conversion (via Crawl4AI)
• System:
  • 🔄 Bull queue system for robust job management (separate queues for Puppeteer & Crawl4AI)
  • 📊 MongoDB for job persistence, status tracking, and results storage
  • 💾 Local file storage for generated assets (screenshots, PDFs, Markdown files)
  • 📈 API endpoints for job management and queue monitoring

Key Technologies

• Backend: Node.js, Express.js
• Web Automation: Puppeteer
• Crawling & AI: Python, FastAPI, Crawl4AI
• Job Queue: BullMQ, Redis
• Database: MongoDB (with Mongoose)
• Language: JavaScript, Python

Installation

Prerequisites

• Node.js (v18 or later recommended)
• npm or yarn
• Python (v3.8 or later recommended)
• pip
• MongoDB (local instance or Atlas)
• Redis (local instance or cloud provider)

Setup

1. Clone the repository:

   git clone <repository-url>
   cd PuppetMaster
   

2. Install Node.js dependencies:

   npm install
   # or