cheap-RAG Development Roadmap

Overview

This roadmap outlines planned enhancements to transform cheap-RAG from a functional document retrieval system into a production-ready, state-of-the-art RAG framework. Priorities are based on impact vs. effort analysis and alignment with mainstream RAG best practices.

---

Phase 1: Core RAG Completion & Production Readiness (Q1 2025)

Goal: Complete missing core functionality and ensure production stability

1.1 Generation Integration (Week 1-2)

Priority: 🔴 Critical Status: Not Started

Tasks:

• Create /api/v1/rag/complete endpoint for full RAG pipeline
• Implement context formatting from retrieved chunks
• Add streaming response support for LLM generation
• Implement citation tracking (link answers to source chunks)
• Add configurable prompt templates
• Support multiple LLM backends (OpenAI, Anthropic, local models)

Deliverables:

# Example endpoint
@router.post('/rag/complete')
async def rag_complete(request: RAGRequest):
    """Full RAG: Retrieval + Generation"""
    chunks = await retrieve(request.query, request.domain)
    answer = await generate(chunks, request.query)
    return {"answer": answer, "sources": chunks, "citations": [...]}

Acceptance Criteria:

• End-to-end RAG workflow functional
• Citations traceable to source documents
• Response time < 3s for typical queries

---

1.2 Advanced Reranking (Week 3-4)

Priority: 🔴 Critical Status: Not Started

Tasks:

• Implement Reciprocal Rank Fusion (RRF) to replace naive cross-sort
• Add cross-encoder reranker (bge-reranker-v2-m3 or Cohere)
• Create reranker abstraction for pluggable models
• Add reranker performance benchmarks
• Make reranking optional via config flag

Technical Details:

# cheap_rag/modules/workflow/workflow_paper/query_workflow/reranker.py
class RRFReranker:
    def rerank(self, results_lists: List[List[dict]], k: int = 60) -> List[dict]:
        """Reciprocal Rank Fusion"""

class CrossEncoderReranker:
    def __init__(self, model_name: str = "BAAI/bge-reranker-v2-m3"):
        """Cross-encoder reranking for top-k refinement"""

Expected Impact:

• 20-30% improvement in retrieval relevance (NDCG@10)
• Better handling of semantic vs. keyword matching conflicts

---

1.3 Metadata Filtering (Week 5)

Priority: 🟡 High Status: TODO exists in code (query_workflow/workflow_tasks.py:64)

Tasks:

• Design metadata schema (date, source, author, tags, etc.)
• Update document insertion to accept metadata
• Implement metadata filtering in Milvus search
• Implement metadata filtering in Elasticsearch search
• Add filter DSL to /search API
• Document filter syntax and examples

API Example:

POST /api/v1/query/search
{
  "query": "machine learning trends",
  "domain": "research_papers",
  "filter": {
    "date_range": {"gte": "2024-01-01", "lte": "2024-12-31"},
    "source": ["arxiv", "pubmed"],
    "tags": {"any": ["ai", "ml"]}
  }
}

---

1.4 Production Hardening (Week 6-8)

Priority: 🟡 High Status: Partial (basic error handling exists)

Tasks:

• Error Handling:

  • Replace generic exceptions with specific types
  • Add retry logic with exponential backoff (tenacity library)
  • Implement circuit breakers for external services
  • Structured error responses with error codes

• Observability:

  • Add Prometheus metrics (query latency, throughput, error rates)
  • Implement structured logging (query_id tracking)
  • Add health check endpoints for dependencies (Milvus, ES, MinIO)
  • Create Grafana dashboard templates

• Rate Limiting:

  • Per-user/per-IP rate limiting (slowapi)
  • Token bucket for embedding API calls
  • Queue management for bulk operations

• Testing:

  • Unit tests for core modules (target: 70% coverage)
  • Integration tests for workflow pipelines
  • Load testing scenarios (locust/k6)

Metrics to Track:

# Prometheus metrics
rag_queries_total (counter)
rag_query_latency_seconds (histogram)
rag_retrieval_results (histogram)
rag_errors_total (counter by type)
embedding_cache_hit_rate (gauge)

---

Phase 2: Performance & Cost Optimization (Q2 2025)

Goal: Reduce latency and operational costs without sacrificing quality

2.1 Intelligent Caching (Week 9-10)

Priority: 🟡 High Status: Not Started

Tasks:

• Embedding Cache:

  • In-memory LRU cache for query embeddings
  • Redis backend for distributed caching
  • Cache warming strategies
  • TTL and invalidation policies

• Result Cache:

  • Cache full retrieval results (query hash → chunks)
  • Configurable TTL based on collection update frequency
  • Cache-aside pattern with automatic refresh

• Metadata Cache:

  • Cache file name mappings (MD5 → original name)
  • Cache collection schemas

Expected Impact:

• 5-10x speedup for repeated queries
• 60-80% reduction in embedding API calls
• Lower infrastructure costs

Implementation:

# cheap_rag/modules/cache/
├── embedding_cache.py
├── result_cache.py
└── metadata_cache.py

---

2.2 Batch Processing & Concurrency (Week 11)

Priority: 🟢 Medium Status: Partial (async exists, batching incomplete)

Tasks:

• Batch query embeddings (multiple queries → single embedding call)
• Implement request coalescing for identical queries
• Optimize Elasticsearch multi-search batch sizes
• Add connection pooling configuration tuning
• Implement worker auto-scaling based on queue depth

Expected Impact:

• 3-5x throughput improvement under load
• Better resource utilization

---

2.3 Model Optimization (Week 12)

Priority: 🟢 Medium Status: Not Started

Tasks:

• Evaluate smaller embedding models for latency-sensitive queries
• Implement model quantization (ONNX int8)
• Support Matryoshka embeddings (variable dimensions)
• Binary/int8 vector search in Milvus
• A/B testing framework for model swaps

Models to Evaluate:

• Current: multilingual-e5-large-instruct (560M params, 1024-dim)
• Alternatives:
  • bge-small-en-v1.5 (33M params, 384-dim) - 5x faster
  • gte-small (lightweight)
  • Custom fine-tuned models

---

Phase 3: Advanced Retrieval Features (Q2-Q3 2025)

Goal: Implement state-of-the-art retrieval techniques

3.1 Query Enhancement (Week 13-15)

Priority: 🟢 Medium Status: Not Started

Tasks:

• HyDE (Hypothetical Document Embeddings):

  • Generate hypothetical answer with LLM
  • Embed hypothetical answer instead of query
  • Configurable via API parameter

• Multi-Query Expansion:

  • Generate query variations with LLM
  • Parallel retrieval for each variation
  • Result fusion and deduplication

• Query Decomposition:

  • Break complex queries into sub-queries
  • Sequential or parallel sub-query execution
  • Result synthesis

• Step-Back Prompting:

  • Ask broader conceptual questions first
  • Use results to guide detailed retrieval

Configuration:

# cheap_rag/modules/workflow/workflow_paper/config.py
query_enhancement:
  enabled: true
  strategies: ["hyde", "multi_query"]
  hyde:
    llm_model: "gpt-4o-mini"
    temperature: 0.7
  multi_query:
    num_variations: 3

---

3.2 Result Diversification (Week 16)

Priority: 🟢 Medium Status: Not Started

Tasks:

• Implement Maximal Marginal Relevance (MMR)
• Add diversity parameter to API
• Support clustering-based diversification
• Time-based diversity (spread across documents/dates)

API Enhancement:

@router.post('/search')
async def search(request: DataSearchEngineRequest):
    result = await QUERY_WORKFLOW.search(
        query=request.query,
        domain=request.domain,
        topk=request.topk,
        diversity=request.diversity,  # 0.0-1.0, 0=pure relevance, 1=max diversity
        diversity_method=request.diversity_method  # "mmr" | "clustering"
    )

---

3.3 Conversational RAG (Week 17-19)

Priority: 🟢 Medium Status: Not Started

Tasks:

• Add session management (conversation history)
• Implement query rephrasing with conversation context
• Support follow-up questions
• Memory compression for long conversations
• Conversational endpoints (/chat/start, /chat/message, /chat/end)

Architecture:

# cheap_rag/modules/conversation/
├── session_manager.py    # Redis-backed session storage
├── query_rewriter.py     # Rewrite queries with chat history
└── memory.py             # Conversation memory management

---

Phase 4: Content Understanding (Q3 2025)

Goal: Better extraction and understanding of multimodal content

4.1 Semantic Chunking (Week 20-21)

Priority: 🟢 Medium Status: Not Started (current: fixed-size chunking)

Tasks:

• Implement semantic boundary detection
• Use embedding similarity for chunk splitting
• LLM-based agentic chunking
• Proposition-based chunking
• A/B test against current token-based chunking

Expected Impact:

• More coherent chunks
• Better retrieval precision
• Reduced context loss at boundaries

---

4.2 Enhanced Table Understanding (Week 22-23)

Priority: 🟢 Medium Status: Partial (basic extraction exists)

Tasks:

• Enable table Q&A generation (build_table_qa=True)
• Table summarization with LLM
• Table-to-text natural language conversion
• Column type detection and schema extraction
• Table chain-of-thought reasoning

---

4.3 Vision & Multimodal (Week 24-26)

Priority: 🔵 Low Status: Basic (URL + caption only)

Tasks:

• Integrate vision LLM (GPT-4V, LLaVA, Qwen-VL)
• Generate image descriptions and Q&A pairs
• CLIP embeddings for image similarity search
• OCR for text extraction from images
• Chart/diagram understanding
• Multimodal retrieval (text + image queries)

---

Phase 5: Advanced Features (Q4 2025)

Goal: Cutting-edge RAG capabilities

5.1 Knowledge Graph Integration (Week 27-30)

Priority: 🔵 Low Status: Not Started

Tasks:

• Entity extraction from documents
• Relation extraction
• Knowledge graph construction (Neo4j)
• Graph-based retrieval (graph RAG)
• Hybrid: Vector + BM25 + Graph

---

5.2 Retrieval-Augmented Fine-tuning (Week 31-34)

Priority: 🔵 Low Status: Not Started

Tasks:

• Collect query-document relevance feedback
• Fine-tune embedding models on domain data
• Hard negative mining
• Continuous learning pipeline
• Model versioning and rollback

---

5.3 Contextual Compression (Week 35-36)

Priority: 🔵 Low Status: Not Started

Tasks:

• Retrieve large candidate set (topk=50)
• Extract relevant sentences/passages
• Compress with extractive/abstractive methods
• Return compressed context to LLM

---

Phase 6: Evaluation & Iteration (Ongoing)

Goal: Continuous improvement through measurement

6.1 Evaluation Framework (Week 37-40)

Priority: 🟡 High Status: Not Started

Tasks:

• Dataset Creation:

  • Golden query-answer pairs
  • Relevance judgments for retrieval
  • Adversarial/edge cases

• Metrics Implementation:

  • Retrieval: NDCG@k, MRR, Recall@k
  • Generation: BLEU, ROUGE, BERTScore
  • RAG-specific: Answer relevance, faithfulness, context precision
  • Ragas integration

• Automated Testing:

  • Regression tests on golden set
  • A/B testing framework
  • Champion/challenger model evaluation

---

Milestones & Success Metrics

Milestone 1: Production-Ready Core (End of Phase 1)

• ✅ Full RAG generation available
• ✅ 99.9% API uptime
• ✅ P95 latency < 2s
• ✅ Error rate < 0.1%
• ✅ 70% test coverage

Milestone 2: Performance Optimized (End of Phase 2)

• ✅ 80% cache hit rate for common queries
• ✅ 5x throughput improvement vs. baseline
• ✅ 50% reduction in infrastructure costs

Milestone 3: SOTA Retrieval (End of Phase 3)

• ✅ NDCG@10 > 0.85 on benchmark
• ✅ Support for 5+ query enhancement strategies
• ✅ Conversational RAG functional

Milestone 4: Multimodal Excellence (End of Phase 4)

• ✅ Table understanding accuracy > 90%
• ✅ Image Q&A functional
• ✅ Semantic chunking shows 10%+ relevance improvement

Milestone 5: Advanced Capabilities (End of Phase 5)

• ✅ Knowledge graph retrieval operational
• ✅ Domain-tuned embedding models deployed
• ✅ Contextual compression reduces token usage by 40%

---

Risk Mitigation

Technical Risks

• Model size/latency trade-off: Use tiered models (fast small model → accurate large model)
• Cache invalidation complexity: Implement conservative TTLs, manual invalidation API
• Backward compatibility: Version API endpoints, support legacy formats

Operational Risks

• Resource constraints: Implement auto-scaling, queue management
• Dependency failures: Circuit breakers, fallback strategies
• Data quality: Validation pipelines, quality scoring

---

Resource Requirements

Team

• Phase 1-2: 2 backend engineers (full-time)
• Phase 3-4: +1 ML engineer (full-time)
• Phase 5: +1 research engineer (part-time)

Infrastructure

• Current: Milvus + ES + MinIO (single node)
• Phase 2: Redis cache, load balancer
• Phase 3: Multi-node clusters, GPU for reranking
• Phase 5: Neo4j, model training infrastructure

Budget (Estimated)

• Phase 1: $5K/month (API costs, compute)
• Phase 2: $8K/month (+caching, scaling)
• Phase 3: $12K/month (+GPU, LLM calls)
• Phase 4-5: $15K/month (+vision models, graph DB)

---

Review & Adjustment

This roadmap will be reviewed quarterly with adjustments based on:

• User feedback and feature requests
• Performance metrics and bottlenecks
• Industry trends and new research
• Resource availability and priorities

Next Review: End of Q1 2025

---

Contributing

For questions or suggestions about this roadmap, please:

1. Open an issue in the repository
2. Tag with roadmap label
3. Provide use case and impact analysis

Last Updated: 2025-10-22