AI Agent Architecture for Croner App

Overview

This document outlines a scalable, provider-agnostic AI agent infrastructure for the Croner App Admin Portal. The architecture is designed to be flexible enough to support both OpenAI and Google Vertex AI, with the ability to expand to additional use cases over time.

Architecture Diagrams

1. High-Level AI Agent Architecture

Complete AI system showing:

• Admin Portal: AI-enhanced UI components (chat, insights, survey assistant, data cleaner)
• AI Service Layer: Agent orchestrator with specialized agents (Data Analyst, Survey Expert, Data Cleaner, Chat Assistant)
• Provider Abstraction: Strategy pattern supporting both OpenAI and Vertex AI
• Vector Database: RAG (Retrieval Augmented Generation) for grounded responses
• Monitoring & Observability: Cost tracking, metrics, distributed tracing

Key Components:

• ✅ Provider-agnostic design (easy to switch between OpenAI/Vertex)
• ✅ Specialized agents for different use cases
• ✅ RAG integration for reducing hallucinations
• ✅ Built on existing infrastructure (Django, Celery, PostgreSQL)

2. AI Agent Interaction Flows

Four detailed sequence diagrams:

Use Case 1: AI Data Analysis

• Admin requests AI insights on job data
• System fetches data from PostgreSQL, retrieves similar analyses from vector DB
• LLM generates insights with context
• Results cached in Redis (30min TTL)

Use Case 2: Real-Time AI Chat

• WebSocket-based streaming chat
• Conversation history maintained in Redis
• RAG retrieves relevant documentation
• Streaming response for real-time UX

Use Case 3: AI Survey Question Generation

• Admin provides survey topic
• System retrieves similar surveys and best practices
• LLM generates structured questions (JSON output)
• Questions saved as drafts for editing

Use Case 4: Async AI Data Cleaning

• CSV upload triggers Celery task
• Data processed in batches (1000 rows)
• LLM validates, cleans, and standardizes data
• Email notification on completion

3. AI Provider Abstraction Layer

Strategy Pattern for provider flexibility:

Interface Methods:

class AIProviderInterface:
    def generate_completion(prompt, model, params)
    def generate_embeddings(texts)
    def stream_response(prompt, callback)
    def function_calling(prompt, functions)
    def get_token_count(text)

Implementations:

• OpenAIProvider: GPT-4o, GPT-4-turbo, text-embedding-3-large
• VertexAIProvider: Gemini 1.5 Pro (2M context), Gemini Flash, text-embedding-004
• Future: Anthropic (Claude 3.5), Local (Llama 3.1)

Provider Router:

• Cost optimization (select cheaper model for simple tasks)
• Latency requirements (real-time vs batch)
• Context window needs (large data analysis)
• Fallback strategy (primary → secondary)

Monitoring:

• Token consumption tracking
• Cost per request (USD)
• API latency (P50, P95, P99)
• Error rates by provider
• Provider distribution analytics

4. RAG (Retrieval Augmented Generation) Pipeline

Reduces AI hallucinations by grounding responses in real data:

Data Sources:

• Survey templates and questions
• Historical job data (JSONB)
• Documents from Azure Blob
• Variables and survey schema

ETL Pipeline (Celery scheduled jobs):

1. Extract data from PostgreSQL/Azure
2. Chunk text semantically (not fixed size)
3. Clean and normalize data
4. Enrich with metadata (tags, categories, dates)

Embedding Generation:

• Batch processing for efficiency
• Separate collections for different data types
• Dimensions: 1536 (OpenAI) or 3072 (Vertex)

Vector Database:

• Option 1: Pinecone (managed, cloud-native)
• Option 2: Qdrant (self-hosted in Kubernetes)
• Indexes: HNSW (fast approximate) or IVF (large-scale)

Query Pipeline:

1. Query rewriter (optimization)
2. Query expander (synonyms, related terms)
3. Query embedding generation
4. Vector search (semantic similarity)
5. Hybrid search (vector + keyword)
6. MMR reranking (diversity + relevance)
7. Cross-encoder reranker (precision refinement)

Context Assembly:

• Top-K results (k=5-10)
• Metadata filtering (date, survey type)
• System instructions (role, guidelines)
• User context (admin profile, history)

Feedback Loop:

• User feedback (thumbs up/down)
• Relevance metrics analytics
• Periodic embedding retraining

5. AI Deployment Stack

Production-ready Kubernetes deployment:

NEW Services:

• AI Service Pods: FastAPI-based, autoscaling (2-5 replicas)
  • AI Orchestrator (agent router)
  • Agent Runtime (LangChain/LlamaIndex)
  • Provider Clients (OpenAI + Vertex SDK)

Integration with Existing Infrastructure:

• Django API calls AI Service via internal gRPC/REST
• Celery workers handle long-running AI tasks
• Redis for conversation history and response caching
• PostgreSQL for AI logs and metadata
• Azure Blob for documents and exports

Security:

• HashiCorp Vault for API key management
• Azure Key Vault integration (existing)
• Web Application Firewall with rate limiting
• Service mesh (Istio/Linkerd) for traffic control

Observability (NEW):

• Prometheus: Metrics collection (tokens, latency, cost)
• Grafana: Dashboards for AI performance
• Loki: Log aggregation for AI service
• Jaeger: Distributed tracing across services
• LangSmith/LangFuse: LLM-specific observability (prompt chains, token usage)

CI/CD:

• GitHub Actions or Azure DevOps
• Docker Registry for container images
• ArgoCD for GitOps deployment
• Separate deployment pipeline for AI service

---

Technology Stack

AI Frameworks

Component	Technology	Purpose
Agent Framework	LangChain / LlamaIndex	Agent orchestration, chains, memory
API Framework	FastAPI	High-performance async API
Vector DB	Pinecone / Qdrant	Embeddings storage & search
LLM Providers	OpenAI, Vertex AI	Language model inference
Observability	LangSmith / LangFuse	LLM monitoring & debugging

OpenAI Models

• GPT-4o: Fast, cost-effective (recommended for most use cases)
• GPT-4-turbo: Large context window (128k tokens)
• text-embedding-3-large: High-quality embeddings (3072 dimensions)

Google Vertex AI Models

• Gemini 1.5 Pro: Massive context (2M tokens), multimodal
• Gemini 1.5 Flash: Fast responses, lower cost
• text-embedding-004: Enterprise-grade embeddings

---

Implementation Phases

Phase 1: Foundation (Weeks 1-4)

Goal: Set up core AI infrastructure

• Deploy FastAPI AI service as Kubernetes pod
• Implement provider abstraction layer (OpenAI + Vertex)
• Set up vector database (Pinecone or Qdrant)
• Create basic agent orchestrator
• Implement API key management (Vault)
• Set up monitoring (Prometheus + Grafana)
• Deploy LangSmith for LLM observability

Deliverables:

• AI service accepting requests from Django
• Provider router dynamically selecting OpenAI/Vertex
• Basic health checks and metrics

Phase 2: RAG Pipeline (Weeks 5-7)

Goal: Build retrieval augmented generation

• Create ETL pipeline for embedding generation
• Index surveys, job data, documents in vector DB
• Implement semantic search with reranking
• Build context assembly logic
• Add caching layer (Redis)
• Create feedback collection system

Deliverables:

• Vector search returning relevant context
• RAG-enhanced responses with citations
• Scheduled jobs for re-indexing data

Phase 3: Specialized Agents (Weeks 8-12)

Goal: Implement use-case-specific agents

Agent 1: Data Analyst Agent

• Build prompt templates for insights generation
• Integrate with JobDataResults (JSONB)
• Create structured output schemas (trends, outliers)
• Add visualization data generation
• Implement admin UI component

Agent 2: Survey Expert Agent

• Build question generation prompts
• Integrate with survey templates and variables
• Create validation for generated questions
• Add draft saving functionality
• Implement admin UI component

Agent 3: Data Cleaning Agent

• Build validation and cleaning prompts
• Integrate with JobDataOriginal pipeline
• Implement batch processing (Celery)
• Create cleaning report generation
• Add admin UI component

Agent 4: Chat Assistant Agent

• Build conversational prompts
• Implement conversation memory (Redis)
• Add WebSocket streaming
• Create chat UI component
• Implement context-aware responses

Deliverables:

• Four specialized agents operational in admin portal
• UI components integrated with React 18
• End-to-end workflows tested

Phase 4: Production Hardening (Weeks 13-16)

Goal: Production-ready AI system

• Implement rate limiting per user/agent
• Add cost budgets and alerts
• Create fallback strategies (primary → secondary provider)
• Implement retry logic with exponential backoff
• Add comprehensive error handling
• Create admin dashboard for AI usage
• Implement PII detection and redaction
• Add audit logging for compliance
• Performance testing and optimization
• Security audit and penetration testing

Deliverables:

• Production-ready AI service with 99.9% uptime
• Cost controls and monitoring
• Security and compliance measures
• Admin analytics dashboard

Phase 5: Expansion (Future)

Goals: Scale to additional use cases

• Client portal AI features (if needed)
• Additional agents (e.g., Report Generator, Email Composer)
• Multi-language support
• Voice input/output
• Advanced analytics (sentiment, classification)

---

Cost Estimation

OpenAI Pricing (as of 2024)

Model	Input (per 1M tokens)	Output (per 1M tokens)
GPT-4o	$2.50	$10.00
GPT-4-turbo	$10.00	$30.00
text-embedding-3-large	$0.13	-

Vertex AI Pricing (as of 2024)

Model	Input (per 1M tokens)	Output (per 1M tokens)
Gemini 1.5 Pro	$1.25	$5.00
Gemini 1.5 Flash	$0.075	$0.30
text-embedding-004	$0.025	-

Monthly Estimate (100 admin users, moderate usage)

Component	Estimated Cost
LLM API calls (mixed)	$500 - $1,500
Embeddings generation	$50 - $150
Vector DB (Pinecone)	$70 - $200
Redis cache	$50 - $100
Monitoring (LangSmith)	$50 - $150
Additional compute (K8s)	$100 - $300
Total	$820 - $2,400/month

Cost Optimization:

• Use GPT-4o (cheaper) for most tasks, GPT-4-turbo only for large context
• Cache responses in Redis (30min TTL)
• Use Vertex AI for batch processing (50% cheaper)
• Implement smart routing (cost vs latency)

---

Security Considerations

API Key Management

• Store keys in HashiCorp Vault or Azure Key Vault
• Rotate keys monthly
• Use separate keys for dev/staging/production
• Implement key usage monitoring

Data Privacy

• PII Detection: Scan inputs for sensitive data (SSN, credit cards)
• Data Masking: Redact PII before sending to LLM
• Audit Logging: Log all AI interactions (GDPR/SOC2)
• Data Residency: Use Vertex AI if data must stay in specific regions

Rate Limiting

• Per-user limits (e.g., 100 requests/hour)
• Per-agent limits (e.g., 1000 requests/day)
• Cost budgets (e.g., $500/day)
• Circuit breakers for API failures

Prompt Injection Prevention

• Validate user inputs
• Use system message boundaries
• Implement output validation
• Monitor for suspicious patterns

---

Monitoring & Alerts

Key Metrics

• Token Usage: Input/output tokens by agent, user, provider
• Latency: P50, P95, P99 response times
• Cost: Real-time spending by agent/user
• Error Rate: 4xx, 5xx errors by provider
• Cache Hit Rate: Redis cache effectiveness
• Embeddings: Vector DB query latency

Alerts

• Cost exceeds daily budget ($X/day)
• Error rate > 5% for 5 minutes
• Latency P95 > 10 seconds
• API key approaching rate limit
• Vector DB query failures

Dashboards

• Executive Dashboard: Total cost, usage trends, ROI metrics
• Operations Dashboard: Latency, errors, uptime by service
• Agent Dashboard: Usage by agent type, performance metrics
• Cost Dashboard: Spending by provider, agent, user

---

Success Metrics

Technical Metrics

• API response time: < 3 seconds (P95)
• Uptime: > 99.9%
• Error rate: < 1%
• Cache hit rate: > 60%

Business Metrics

• Adoption: % of admins using AI features weekly
• Efficiency: Time saved per task (e.g., survey creation -50%)
• Quality: Accuracy of AI insights (human validation)
• Satisfaction: NPS score for AI features

Cost Metrics

• Cost per request: < $0.10
• ROI: Admin time saved vs AI cost
• Budget adherence: < 10% variance

---

FAQ

Why FastAPI for AI service instead of Django?

FastAPI offers:

• Async support for LLM streaming
• Better performance (3x faster than Django)
• Native WebSocket support
• Smaller container size (faster deployments)
• Auto-generated OpenAPI docs

Django remains for core business logic; FastAPI handles AI-specific workloads.

Why both OpenAI and Vertex AI?

• Redundancy: Fallback if one provider has outages
• Cost optimization: Route tasks to cheaper provider
• Compliance: Vertex AI for data residency requirements
• Feature parity: Some features only available on one provider
• Negotiation leverage: Multi-provider reduces lock-in

Why RAG instead of fine-tuning?

• Cost: RAG is ~10x cheaper than fine-tuning
• Flexibility: Update knowledge base without retraining
• Transparency: Citations show data sources
• Accuracy: Grounded in real data, less hallucination
• Maintenance: No model retraining pipeline needed

Can we use open-source models (Llama, Mistral)?

Yes, the abstraction layer supports it! But consider:

• Infrastructure cost: Need GPU instances ($500-2000/month)
• Performance: Open models lag behind GPT-4/Gemini
• Maintenance: Model updates, optimization, monitoring
• Best for: High-volume, privacy-critical use cases

Start with OpenAI/Vertex, migrate to open-source if scale justifies it.

How do we handle AI hallucinations?

• RAG: Ground responses in real data
• Structured outputs: JSON schemas enforce format
• Validation: Post-process outputs with business rules
• Citations: Show data sources for verification
• Human-in-the-loop: Admin reviews before applying changes
• Feedback loop: Learn from corrections

---

Next Steps

1. Review architecture with engineering team
2. Choose vector DB: Pinecone (managed) vs Qdrant (self-hosted)
3. Decide AI provider: OpenAI, Vertex, or both
4. Prioritize agents: Which use case first?
5. Allocate budget: $1-3k/month for initial rollout
6. Set up dev environment: AI service + vector DB
7. Begin Phase 1: Foundation infrastructure

Estimated Timeline: 16 weeks to production-ready AI system

Team Requirements:

• 1x ML Engineer (AI service, agents, RAG)
• 1x Backend Engineer (Django integration, APIs)
• 1x Frontend Engineer (React UI components)
• 1x DevOps Engineer (K8s deployment, monitoring)

---

Resources

Documentation

• OpenAI API Docs
• Vertex AI Docs
• LangChain Docs
• Pinecone Docs
• LangSmith Docs

Example Implementations

• LangChain + FastAPI Template
• OpenAI Cookbook
• RAG Tutorial

---

Ready to build the future of compensation analytics with AI! 🚀