Gemini Voice Agent - Full-Stack Application

Project Overview

A production-ready full-stack voice AI application that enables real-time voice conversations with Google's Gemini AI model. Features separate admin and user portals with proper backend server, database integration, and WebSocket-based voice streaming.

Architecture

Frontend (React + TypeScript + Vite)

• Framework: React 19 with TypeScript
• Build Tool: Vite 6
• Routing: React Router with separate routes for /voiceagent and /admin
• Port: 5000 (configured for Replit webview)
• API Communication: Proxied to backend via Vite proxy

Backend (Node.js + Express + TypeScript)

• Framework: Express.js with TypeScript
• Port: 3001
• Database: PostgreSQL (Neon) with Drizzle ORM
• WebSocket: Secure Gemini AI voice proxy
• Authentication: JWT-based admin authentication

Database Schema

• users: User accounts (name, phone)
• stages: Voice agent stages with order and validation questions
• stage_prompts: Junction table linking stages to system prompts
• stage_knowledge_items: Junction table linking stages to knowledge base documents
• user_stage_progress: Tracks user progress through stages with context
• system_prompts: AI system prompts with active state
• knowledge_base: Document uploads for AI context
• interaction_logs: Complete conversation logs with transcripts
• app_settings: Application settings (language preference)
• admin_users: Admin authentication credentials

Key Features

Admin Portal (/admin)

• JWT-protected authentication
• Stage Management: Create, edit, delete, and reorder conversation stages
  • Assign system prompts to each stage
  • Attach knowledge base documents to specific stages
  • Define validation questions for stage progression
• Manage system prompts and activate specific prompts
• Upload and manage knowledge base documents
• View all user interaction logs with full transcripts
• Configure application language settings
• Admin Credentials: Set up via npm run add-admin script (no default credentials for security)

Voice Agent Portal (/voiceagent)

• Simple login with name and phone number
• Stage-Based Conversations: Users progress through configured stages
  • Each stage has its own system prompts and knowledge base
  • Agent behavior changes based on current stage
  • Linear progression with validation questions
  • Context maintained across stages
• Real-time voice interaction with Gemini AI
• Live transcription of both user and agent speech
• Automatic conversation logging
• Default language: Telugu-English mix (70% Telugu, 30% English)

Project Structure

/
├── backend/                 # Backend server
│   ├── src/
│   │   ├── api/            # API route handlers
│   │   │   ├── auth.ts     # Authentication endpoints
│   │   │   ├── stages.ts   # Stage management (CRUD, assign prompts/knowledge)
│   │   │   ├── userStage.ts # User stage progression APIs
│   │   │   ├── prompts.ts  # System prompts management
│   │   │   ├── knowledge.ts # Knowledge base management
│   │   │   ├── logs.ts     # Interaction logs
│   │   │   └── settings.ts # App settings
│   │   ├── config/
│   │   │   ├── database.ts # Database connection
│   │   │   └── schema.ts   # Drizzle schema definitions
│   │   ├── services/
│   │   │   └── geminiProxy.ts # WebSocket proxy for Gemini
│   │   ├── middleware/
│   │   │   └── auth.ts     # JWT authentication middleware
│   │   ├── types/
│   │   │   └── index.ts    # TypeScript type definitions
│   │   ├── scripts/
│   │   │   ├── seed.ts     # Database seeding (app settings)
│   │   │   ├── seedStages.ts # Stage initialization script
│   │   │   └── addAdmin.ts # Create admin user
│   │   └── index.ts        # Express server entry point
│   ├── package.json
│   └── tsconfig.json
├── components/             # React components
│   ├── AdminPortal.tsx    # Admin dashboard
│   ├── AdminLogin.tsx     # Admin authentication
│   ├── StageManagement.tsx # Stage management UI
│   ├── Login.tsx          # User login
│   ├── UserPortal.tsx     # Voice interaction UI
│   └── icons/             # Icon components
├── services/              # Frontend services
│   ├── api.ts            # Backend API client
│   └── geminiService.ts  # WebSocket client for voice
├── App.tsx               # Main app with routing
├── types.ts              # Shared TypeScript types
├── vite.config.ts        # Vite configuration with proxy
└── package.json          # Root package.json

## Environment Variables
- `GEMINI_API_KEY`: Required for Google Gemini API access (backend only, secure)
- `DATABASE_URL`: PostgreSQL connection string (auto-configured in Replit)
- `JWT_SECRET`: JWT signing secret (optional, has default)

## Development

### Running Locally
```bash
npm run dev

This starts both frontend (port 5000) and backend (port 3001) concurrently.

Database Management

cd backend
npm run db:push        # Push schema changes to database
npm run db:seed        # Seed initial application settings
npm run db:seed-stages # Create default stages from existing prompts
npm run add-admin      # Create a new admin user (interactive)

Vite Configuration

• Binds to 0.0.0.0:5000 (required for Replit)
• Allows all hosts (required for Replit's proxy/iframe setup)
• Proxies /api/* requests to backend at localhost:3001
• Proxies /voice WebSocket to backend WebSocket server

Security Features

• Gemini API key secured on backend only (never exposed to frontend)
• WebSocket proxy prevents direct client-to-Gemini connections
• JWT-based admin authentication with 24-hour expiration
• Password hashing with bcrypt
• Protected admin routes

Data Flow

User Voice Interaction Flow

1. User logs in via frontend → Backend creates/retrieves user → Returns user object with JWT token
2. Backend checks user's current stage (creates initial stage progress if first-time user)
3. User starts voice session → WebSocket connects through Vite proxy to backend
4. Backend fetches stage-specific prompts and knowledge base documents for current stage
5. Backend creates Gemini session with combined system prompts and knowledge context
6. Audio flows: Browser → Backend → Gemini → Backend → Browser
7. Transcripts and context saved to user_stage_progress in real-time
8. Stage validation: User can advance to next stage upon meeting validation criteria

Admin Management Flow

1. Admin logs in → JWT-protected access to admin portal
2. Admin can create/edit stages with validation questions
3. Admin assigns system prompts and knowledge documents to each stage
4. Changes take effect immediately for new sessions
5. Admin can view all user interactions including stage progression history

Production Database Setup

If you're seeing login errors in production, your production database needs to be populated with data from development.

Quick Fix (Recommended):

1. Open setup-production.md for detailed instructions
2. Or use Replit's database snapshot:
   • Tools → Database → Create Snapshot (on dev)
   • Switch to Production → Restore from Snapshot

Manual Fix:

# Export development data
cd backend && npm run db:export

# Then in Replit UI: Switch to Production database

# Import to production
cd backend && npm run db:import

Recent Changes

Nov 12, 2025 - SPA Routing Investigation (In Progress)

• Issue: Direct access to /voiceagent returns "Cannot GET /voiceagent" 404
• Root Cause: Complex interaction between Vite proxy, Express backend, and port mappings
• Investigation Findings:
  • Vite dev server running on port 5000 with appType: 'spa'
  • Express backend on port 3001 with production SPA fallback configured
  • Requests to /voiceagent return Express headers even on port 5000
  • Root path / works correctly and loads React app
  • React Router client-side navigation works (clicking links)
  • Only direct URL access to SPA routes fails
• Status: Currently investigating Replit's port forwarding and proxy behavior

Nov 12, 2025 - Fixed Login & Gemini WebSocket Connection

• Fixed backend module loading error: Changed TypeScript module from ES2022 to CommonJS
  • ts-node-dev now loads modules correctly
  • Backend server starts successfully on port 3001
• Fixed login APIs: Both user and admin login endpoints working correctly
  • Development login working perfectly ✅
  • Production requires database migration (see Production Database Setup section above)
• Fixed Gemini WebSocket proxy configuration (CRITICAL FIX):
  • Root cause: Vite proxy regex ^/voice$ only matched /voice exactly, not /voice?userId=4
  • Solution: Changed from '^/voice$' to '/voice' to handle query parameters
  • Changed WebSocket target from ws://localhost:3001 to http://localhost:3001
  • Added changeOrigin: true for proper header forwarding
  • Voice agent now connects successfully to Gemini Live API ✅
  • Added debug logging to geminiService for connection troubleshooting

Nov 11, 2025 - Fixed Production Deployment & SPA Routing

• Fixed production 500 error: Backend now serves frontend static files in production
  • In production, everything runs on single Express server (port 3001)
  • Eliminates need for separate frontend server and complex proxy configuration
  • Frontend /api calls work seamlessly in both dev and production
• Fixed critical routing bug: /voice proxy was matching /voiceagent
  • Solution: Changed proxy config from /voice to ^/voice$ (exact match regex)
  • Added custom SPA fallback plugin for Vite dev server
  • Backend serves index.html for all non-API routes in production
• Result: All routes work with direct URL access in both dev and production
  • Development: Vite (5000) + Backend (3001) with proxy
  • Production: Backend only (3001) serving everything
• Updated TypeScript config to ES2022 for proper ES module support

Nov 11, 2025 - Stage-Based System Implementation

• Removed default admin credentials for improved security (admin/admin123 no longer exists)
• Implemented complete stage-based architecture:
  • Created 5 new database tables: stages, stage_prompts, stage_knowledge_items, user_stage_progress
  • Built comprehensive Admin APIs for stage CRUD operations
  • Built Runtime APIs for user stage progression and validation
  • Updated GeminiProxy to dynamically fetch and use stage-specific prompts/knowledge
• Created Stage Management UI in Admin Portal with intuitive interface for:
  • Creating, editing, deleting stages
  • Assigning prompts and knowledge to stages
  • Managing stage order and validation questions
• Stage seeding script to migrate existing prompts into default stage structure
• Security improvements: All admin credentials must be created via add-admin script

Nov 10, 2025 - Initial Full-Stack Implementation

• Created full monorepo structure with separate backend
• Implemented Express backend with TypeScript
• Set up PostgreSQL database with Drizzle ORM
• Created secure WebSocket proxy for Gemini voice streaming
• Implemented JWT authentication for admin and user portals
• Added React Router with /voiceagent and /admin routes
• Configured Vite proxy for seamless frontend-backend communication
• Set default language to Telugu-English mix
• Deployed with VM configuration for always-on backend

Deployment

• Type: VM (always-on server required for WebSocket)
• Build: Builds both frontend (Vite) and backend (TypeScript compilation)
• Production: Backend serves both frontend static files AND API on port 3001
  • Frontend build output served from dist/ folder
  • All /api/* requests handled by Express routes
  • WebSocket connections on /voice endpoint
  • SPA routing supported for direct URL access
• Development: Separate servers
  • Frontend: Vite dev server on port 5000 with hot reload
  • Backend: ts-node-dev on port 3001
  • Vite proxies /api requests to backend
• Public Access: Port 3001 exposed as port 80 on Replit domain