Cursor Rules: Embedding Flavor

Embedding generation, validation, and troubleshooting workflows

include: .cursorrules.base

Embedding Architecture

• Vector Storage: Neo4j vector database for DUX object embeddings
• Multi-Modal: Support for text, structured data, and metadata embeddings
• Real-Time: Generate embeddings on-demand for new objects
• Batch Processing: Efficient bulk embedding generation

Embedding Models

• Text Embeddings: Semantic search for DUX object content
• Structured Embeddings: Vector representations of JSON schemas
• Metadata Embeddings: Embed tags, relationships, and context
• Hybrid Search: Combine vector similarity with graph relationships

Vector Index Management

• Index Creation: Automatic index creation for new object types
• Index Optimization: Tune parameters for search performance
• Index Maintenance: Regular reindexing and cleanup
• Index Monitoring: Track index health and performance

Embedding Generation Workflow

1. Object Validation: Ensure DUX objects pass schema validation
2. Text Extraction: Extract searchable text from object fields
3. Embedding Creation: Generate vector representations
4. Index Storage: Store embeddings in Neo4j vector indices
5. Metadata Linking: Link embeddings to original objects

Search Patterns

• Semantic Search: Find similar DUX objects by meaning
• Evidence Search: Find objects with similar evidence patterns
• Relationship Search: Find objects connected by common themes
• Temporal Search: Find objects by creation or update time

Quality Assurance

• Embedding Validation: Verify embedding quality and consistency
• Search Accuracy: Test search results against known relationships
• Performance Testing: Measure search speed and accuracy
• A/B Testing: Compare different embedding strategies

Troubleshooting Workflows

• Index Corruption: Detect and repair corrupted vector indices
• Embedding Drift: Monitor for embedding quality degradation
• Search Failures: Debug and fix search query issues
• Performance Issues: Optimize slow search operations

Optimization Strategies

• Dimensionality Reduction: Optimize embedding dimensions for performance
• Batch Processing: Efficient bulk embedding operations
• Caching: Cache frequently accessed embeddings
• Compression: Compress embeddings for storage efficiency

Monitoring & Analytics

• Search Metrics: Track search performance and accuracy
• Embedding Quality: Monitor embedding consistency and relevance
• Index Health: Monitor vector index performance
• Usage Patterns: Analyze search behavior and patterns

Integration Patterns

• API Integration: Embedding generation via REST APIs
• Event-Driven: Generate embeddings on object creation/update
• Scheduled Jobs: Batch embedding generation for large datasets
• Real-Time Updates: Immediate embedding updates for critical objects

Security & Privacy

• Data Sanitization: Remove PII before embedding generation
• Access Control: Secure access to embedding generation APIs
• Audit Trail: Track embedding generation and usage
• Encryption: Encrypt embeddings in transit and at rest

Performance Tuning

• Index Parameters: Optimize Neo4j vector index settings
• Query Optimization: Efficient search query patterns
• Resource Allocation: Optimize CPU and memory usage
• Scaling: Horizontal scaling for high-volume embedding operations