WARP.md

This file provides guidance to WARP (warp.dev) when working with code in this repository.

Overview

This repository is an ASR (Automatic Speech Recognition) evaluation system and media processing toolkit built for:

• Benchmarking multiple ASR models (Whisper variants, Canary-Qwen, OLMoASR, etc.)
• Processing video/audio content with subtitle generation and synchronization
• Managing evaluation datasets with Hugging Face integration
• Running distributed GPU workloads on RunPod cloud infrastructure

The system supports both containerized development (Docker/JupyterLab) and various deployment modes (full evaluation, fast GPU-optimized, minimal CPU-only).

Key Commands

Development Environment

Start containerized environment:

./start.sh                    # Start Docker Compose services with auto-token retrieval
docker compose up --build -d  # Manual Docker Compose startup

Access the environment:

docker compose exec jupyterlab bash  # Shell into container as jovyan user
# Work directory: /home/jovyan/workspaces

Testing and Evaluation

Run ASR model tests:

python run_tests.py           # Basic model availability test
python run_tests.py --full    # Full pytest suite with coverage

Run ASR model evaluations:

python asr_leaderboard.py                           # Full comprehensive evaluation
python asr_leaderboard.py --mode fast              # GPU-optimized fast evaluation  
python asr_leaderboard.py --mode minimal           # CPU-only minimal evaluation
python asr_leaderboard.py --debug                  # Enable detailed logging

Dataset Management

Manage evaluation datasets:

python dataset.py list                             # List available datasets
python dataset.py info vaporeon                    # Get dataset information
python dataset.py get vaporeon --path datasets/    # Download/verify dataset

Architecture Overview

Core Components

ASR Evaluation System (src/providers/asr/):

• core/interfaces.py - Abstract base classes (ASRModelAdapter, TranscriptionResult, ModelInfo)
• adapters/ - Model-specific implementations:
  • faster_whisper_adapter.py - Local Whisper via faster-whisper
  • canary_qwen_adapter.py - NVIDIA Canary-Qwen models via transformers
  • olmoasr_adapter.py - Allen AI OLMoASR models
• core/config.py - Configuration management (ConfigManager loads from config/asr_models.yaml)
• storage/postgres_storage.py - PostgreSQL result persistence

Dataset Management (src/datasets/):

• manager.py - Auto-discovery system for dataset handlers
• base_handler.py - Abstract dataset handler interface
• handlers/ - Dataset-specific handlers (vaporeon, gophers, etc.)

Media Processing (src/):

• subtitle_processor.py - Subtitle generation and formatting
• video_remuxer.py - Video container operations
• process_wmv_files.py - Windows Media Video processing

Infrastructure (src/infra/):

• runpod_manager.py - RunPod GPU cloud management

Configuration System

The system uses a hierarchical configuration approach:

1. Model Configuration (config/asr_models.yaml):

   • Defines all supported ASR models with performance targets
   • Specifies device requirements, use cases, and fallback chains
   • Supports model selection by use case: highest_accuracy, fastest_processing, multilingual, transparent

2. Environment Variables (.env):

   • API keys: OPENROUTER_API_KEY, CLAUDE_API_TOKEN, RUNPOD_API_KEY
   • Hugging Face: HF_TOKEN, HF_USERNAME
   • Database: POSTGRES_* settings

3. Runtime Configuration:

   • Models auto-discovered via ConfigManager.get_models_by_use_case()
   • Fallback chains ensure graceful degradation
   • Processing modes: transcript_only, subtitles, diarization, real_time

Evaluation Flow

1. Model Registration: ConfigManager loads model configs from YAML
2. Adapter Creation: Factory pattern creates appropriate ASRModelAdapter instances
3. Benchmarking: asr_leaderboard.py runs models against reference datasets
4. Metrics Calculation: WER/CER computed via Levenshtein distance
5. Result Storage: Optional PostgreSQL persistence for historical analysis

Development Patterns

Adding New ASR Models

1. Create adapter in src/providers/asr/adapters/new_model_adapter.py:

   • Inherit from ASRModelAdapter
   • Implement transcribe(), get_model_info(), is_available()

2. Add configuration in config/asr_models.yaml:

   • Define model metadata, parameters, device requirements
   • Add to appropriate use case categories

3. Register in evaluation (asr_leaderboard.py):

   • Add to models_to_test list with factory function

Adding New Datasets

1. Create handler in src/datasets/handlers/dataset_name_handler.py:

   • Inherit from BaseDatasetHandler
   • Implement get(), info(), verify()

2. Auto-discovery: DatasetManager will automatically detect and register the handler

Important Rules and Constraints

Security Requirements

• Talisman pre-commit hook is mandatory - scans for secrets/sensitive data
• Never commit API keys - use .env file with environment variables
• No emojis in code or documentation (organization policy)

Development Environment

• Use Docker containers for development to avoid system package conflicts
• Work as jovyan user inside containers (/home/jovyan/workspaces)
• GPU support enabled in Docker Compose for CUDA models

Code Quality

• Run tests before commits: python run_tests.py --full
• Follow Python conventions: snake_case, type hints
• Security-first approach: Talisman will block unsafe commits

Model Performance Targets

Reference from config/asr_models.yaml:

• Excellent WER: < 5% (canary-qwen-2.5b targets 5.63%)
• Good WER: < 10% (faster-whisper-base targets 9.2%)
• Acceptable WER: < 20%

Hardware Considerations

• VRAM requirements vary: tiny models (1GB) to large models (16GB+)
• Fallback chains defined for when preferred models unavailable
• CPU/GPU mode selection automatic based on hardware availability

File Structure Notes

• Notebooks in notebooks/ are git-tracked for reproducible experiments
• Configuration in config/ centralized for model and evaluation settings
• Tests in tests/ with pytest configuration in pytest.ini
• Docker setup supports both minimal (Dockerfile.minimal) and full (Dockerfile) environments
• Media processing logs go to .log files (gitignored for sensitive data)