Search within specific document types

title: Search And Retrieval audience: users status: active owner: product-education last_reviewed: 2025-03-13

Search & Retrieval Guide

Status: Active
Last Updated: 2025-10-01 Purpose: Complete guide to effective searching and information retrieval
Audience: Users who want to find information effectively

Master the art of finding exactly what you need with our AI-enhanced search system. This guide covers everything from basic searches to advanced techniques that leverage artificial intelligence to understand your intent and deliver better results.

How Our Search Works

Our system uses multiple AI techniques working together to understand what you're looking for:

🧠 Intelligent Understanding

Semantic Search: Understands meaning, not just keyword matching
HyDE Compatibility Mode: Provides the historic HyDE entry point while delegating to the unified hybrid search pipeline
Smart Reranking: Uses AI to reorder results by relevance
Multi-Stage Retrieval: Searches broadly first, then refines for precision

🔍 What This Means For You

Find concepts, not just keywords: Search for "database optimization" and find content about "performance tuning"
Natural language queries: Ask questions like "how do I speed up searches?"
Better result quality: Most relevant information appears first
Fewer irrelevant results: AI filters out noise automatically

Getting the Best Results

Writing Effective Queries

✅ Good Query Examples:

"how to optimize vector database performance"
"troubleshooting slow search responses"
"best practices for document indexing"
"web scraping rate limiting strategies"

❌ Avoid These Patterns:

"database" (too vague)
"fix error" (no context)
"help" (not specific)
"documentation" (too broad)

Query Writing Tips

Technique	Example	Why It Works
Be specific	"configure OpenAI embeddings" vs "setup"	Targets exact needs
Include context	"Python web scraping tutorial" vs "scraping"	Adds important details
Use natural language	"Why are my searches slow?"	AI understands intent
Mention the outcome	"reduce memory usage during indexing"	Focuses on goals

Search Strategies by Use Case

📚 Research & Learning

Use question format: "What is the difference between dense and sparse vectors?"
Include learning level: "beginner guide to vector databases"
Ask for comparisons: "Redis vs DragonflyDB performance"

🔧 Troubleshooting

Describe the problem: "search results are empty"
Include error context: "timeout errors during large document indexing"
Mention what you tried: "increased chunk size but still getting errors"

⚙️ Configuration

Be specific about settings: "configure HNSW parameters for accuracy"
Mention your environment: "production deployment configuration"
Include constraints: "optimize for cost with minimal accuracy loss"

Search Features Explained

HyDE Enhancement (Compatibility Mode)

What it does now: The hyde_search MCP tool is retained for backward compatibility, but under the hood it simply calls the consolidated hybrid search path provided by VectorStoreService. No additional hypothetical documents are generated in Phase 3.

When to use it: Reach for hyde_search when you prefer the previous API contract or want a named tool for hybrid search. For new workflows, the default search_documents or filtered_search tools are equivalent in capability.

User experience: Expect faster responses with the same query semantics—the result list mirrors the unified hybrid search pipeline.

Semantic Reranking (Automatic)

What it does: After finding potential matches, AI re-examines each result to determine true relevance to your query.

When it helps most:

Eliminating false positives
Finding nuanced matches
Prioritizing comprehensive answers
Filtering out tangentially related content

User experience: The first few results will be significantly more relevant than traditional keyword-based searches.

Multi-Stage Retrieval (Automatic)

What it does: First finds many potential matches quickly, then applies sophisticated filtering to identify the best ones.

Benefits for users:

Faster initial response
Higher precision in final results
Balances speed with accuracy
Handles large document collections efficiently

Retrieval modes (Configurable)

What it does: Controls which embedding signals power your search.

dense – Uses FastEmbed dense vectors for semantic similarity.
sparse – Uses FastEmbedSparse BM25-style signals for lexical precision.
hybrid (default) – Combines both and lets Qdrant fuse scores during ranking.

How to use it:

REST/MCP payloads: set search_strategy (REST) or retrieval_mode (config/CLI) to dense, sparse, or hybrid.
CLI: pass --retrieval-mode to mcp search, mcp add-documents, or mcp add-documents-batch.

User experience: Hybrid usually provides the strongest relevance, while dense can reduce latency for short-term experiments and sparse helps when chasing exact keyword matches.

Performance Expectations

Search Response Times

Our enhanced database connection pooling delivers significantly improved performance:

Simple queries: < 250ms (50.9% faster than previous versions)
Complex semantic searches: 500ms-1.5 seconds (improved from 1-3 seconds)
HyDE-enhanced queries: 1-2.5 seconds (improved from 2-5 seconds)
Large result sets: 1.5-3.5 seconds (improved from 3-7 seconds)
Peak load performance: Up to 887.9% higher throughput during busy periods

Result Quality Indicators

High relevance: First 3-5 results directly address your query
Good coverage: Results from multiple relevant documents
Varied perspectives: Different approaches to the same topic
Recent information: When available, newer content is prioritized
Consistent performance: Intelligent connection pooling maintains fast response times even during peak usage

Automatic Performance Optimization

The system now includes smart database optimization that:

Scales automatically: Connection pools adjust based on your usage patterns
Monitors performance: Tracks query speeds and optimizes accordingly
Prevents bottlenecks: Intelligent load balancing prevents slowdowns
Recovers gracefully: Circuit breaker patterns handle temporary issues
No configuration needed: Works optimally out of the box

Advanced Search Techniques

Using Search Filters

# Search within specific document types
query: "API documentation" filter: type=reference

# Search recent content only
query: "deployment strategies" filter: date>2024-01-01

# Search specific projects or collections
query: "configuration" filter: project=production

Federated Search (Cross-Collection)

New in this version: Search across multiple collections simultaneously for comprehensive results.

# Search across all available collections
query: "microservices architecture" collections: all

# Search specific collections with ranking
query: "API authentication" collections: documentation,examples,tutorials

# Weighted search across collections
query: "performance optimization" weights: documentation=0.5,benchmarks=0.8,examples=0.3

Benefits:

Comprehensive coverage: Find information across all your data sources
Intelligent ranking: Results are ranked across collections for relevance
Adaptive search modes: System chooses optimal search strategy per collection
Result fusion: Advanced algorithms combine results from multiple sources

Combining Search Types

Broad + Specific: Start broad ("web scraping"), then narrow ("browser automation headless mode")
General + Technical: Begin with concepts ("caching"), then specifics ("DragonflyDB configuration")
Problem + Solution: Search the issue first, then look for implementation approaches

Power User Tips

🎯 Precision Searching

Use quotes for exact phrases: "error code 500"
Include specific versions: "Python 3.13 compatibility"
Mention specific tools: "Qdrant collection aliases"

🔄 Iterative Refinement

Start with a broad search
Review top results for better terminology
Refine your query using discovered keywords
Repeat until you find exactly what you need

📋 Research Workflows

Search for overviews first: "system architecture overview"
Then dive into specifics: "client management implementation"
Cross-reference with examples: "client management code examples"

Troubleshooting Search Issues

No Results Found

Possible causes:

Query too specific or uses uncommon terminology
Documents not indexed yet
Spelling errors in technical terms

Solutions:

Try broader, more general terms
Check if documents are in the system: use collection listing
Use alternative terminology: "config" instead of "configuration"
Break complex queries into simpler parts

Poor Result Quality

Symptoms:

Irrelevant results in top positions
Missing obvious matches
Results don't match query intent