Valdo Backend Copilot Rules

# Copilot Instructions for valdo-backend

## Project Overview
This is a daily data pipeline for scraping HotNewHipHop music releases and enriching them with YouTube and Spotify links. The project uses TypeScript with ESM modules, Express.js for the REST API, and integrates with OpenAI, YouTube Data API, and Spotify Web API.

## Technology Stack
- **Runtime**: Node.js 18+ with ESM module support
- **Language**: TypeScript 5.x with strict mode
- **Framework**: Express.js 5.x
- **Web Scraping**: Cheerio and Axios
- **AI Integration**: OpenAI SDK (GPT-3.5-turbo)
- **APIs**: YouTube Data API, Spotify Web API, Google APIs
- **Development**: tsx for watch mode, tsc for builds

## Project Structure
```
valdo-backend/
├── src/
│   └── server.ts          # Express server entry point
├── routes/
│   └── jobs.ts            # Job routes (POST /jobs/daily-hnhh)
├── services/
│   ├── scraper.ts         # Cheerio scraping logic
│   ├── llm.ts             # OpenAI extraction service
│   ├── youtube.ts         # YouTube Data API integration
│   ├── spotify.ts         # Spotify Web API integration
│   └── pipeline.ts        # Orchestrates the daily workflow
├── storage/
│   └── store.ts           # File-based storage utilities
└── utils/
    └── date.ts            # Date utility functions
```

## Code Style and Conventions

### TypeScript
- Use strict mode TypeScript with all strict options enabled
- Use ESM imports with `.js` extensions (even for TypeScript files)
- Example: `import { func } from '../services/pipeline.js';`
- Define explicit types for function parameters and return values
- Use interfaces for complex objects and data structures
- Prefer `const` over `let`; avoid `var`
- Use async/await over promise chains

### Module System
- **IMPORTANT**: This project uses ESM modules (`"type": "module"` in package.json)
- Always use ESM import/export syntax
- Import TypeScript files with `.js` extension (TypeScript compiles .ts to .js)
- Do NOT use CommonJS (`require`/`module.exports`)
- Do NOT use `import x = require('x')` syntax

### Comments and Documentation
- Use JSDoc comments for exported functions and interfaces
- Include parameter descriptions and return type documentation
- Keep inline comments minimal; code should be self-explanatory
- Use TODO comments for planned improvements with context

### Error Handling
- Use try-catch blocks for async operations
- Log errors to console with descriptive messages
- Return error information in API responses
- Type errors as `any` when catching to access `.message` property
- Example: `catch (error: any) { console.error('...', error); }`

### API Responses
- Use consistent JSON response format with `message` and data fields
- Return appropriate HTTP status codes (200 for success, 500 for errors)
- Include error messages in development mode only
- Use async route handlers with proper error handling

### Service Layer Patterns
- Keep services focused on single responsibilities
- Export async functions for service operations
- Use lazy initialization for API clients (check environment variables)
- Return structured objects with success/error states
- Log progress and errors at service boundaries

### Logging
- Use `console.log` for informational messages
- Use `console.error` for errors
- Include timestamps via middleware for HTTP requests
- Log pipeline steps with descriptive messages
- Format: `` console.log(`Step X: Description...`); ``

### Environment Variables
- Load environment variables using `dotenv.config()`
- Check for required environment variables before use
- Provide fallback behavior when API keys are missing (e.g., mock data)
- Never commit `.env` files; maintain `.env.example` with placeholders

### Async Patterns
- Use `async/await` for asynchronous operations
- Add small delays between API calls to respect rate limits
- Example: `await new Promise((resolve) => setTimeout(resolve, 500));`
- Process arrays sequentially when order matters or rate limits apply
- Use `Promise.all()` for parallel operations when safe

## Build and Development

### Commands
- **Install dependencies**: `npm install`
- **Development mode**: `npm run dev` (uses tsx watch)
- **Build**: `npm run build` (compiles TypeScript to dist/)
- **Production**: `npm start` (runs compiled code from dist/)
- **Test**: Currently not implemented (`npm test` exits with error)

### Build Configuration
- TypeScript compiles to `dist/` directory
- Source maps and declarations are generated
- Target: ES2022
- Module: ESNext with Node resolution
- All source directories included: src, routes, services, storage, utils

### Development Workflow
1. Make changes to TypeScript files
2. tsx watch automatically reloads in dev mode
3. Test changes using API endpoints via curl or HTTP clients
4. Build with `npm run build` before deployment

## API Endpoints

### POST /jobs/daily-hnhh
Triggers the daily HotNewHipHop scraping pipeline
- Optional body: `{ "date": "YYYY-MM-DD" }`
- Defaults to yesterday's date if not provided
- Returns pipeline execution result with track counts

### GET /jobs/history
Get pipeline execution history (currently returns empty array)

### GET /jobs/status
Get current job status (currently returns static "idle" status)

### GET /health
Health check endpoint returning status, timestamp, and uptime

### GET /
Root endpoint returning API documentation and available endpoints

## Common Patterns

### Pipeline Orchestration
- Pipeline steps are sequential and logged
- Each step handles its own errors and returns structured data
- Results are saved to JSON files in the data directory
- Execution is logged to `pipeline-runs.log`

### Data Flow
1. Scrape archive page for track URLs
2. Scrape individual track pages for HTML content
3. Extract structured data using OpenAI
4. Enrich with YouTube links
5. Enrich with Spotify links
6. Save final results to JSON file

### API Client Initialization
- Use singleton pattern for API clients
- Check environment variables before creating clients
- Return mock/default data when API keys are missing
- Example pattern from llm.ts:
  ```typescript
  let openai: OpenAI | null = null;
  
  function getOpenAIClient(): OpenAI {
    if (!openai) {
      openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
    }
    return openai;
  }
  ```

## Known TODOs and Future Improvements
- Implement actual HNHH archive URL structure and selectors
- Optimize HTML content sent to OpenAI (reduce token usage)
- Add concurrent scraping with rate limiting
- Implement job queue for background processing
- Add authentication/authorization
- Add proper error handling and retries
- Implement quota management for API calls
- Add comprehensive logging
- Add unit and integration tests
- Add metrics and monitoring

## Important Notes
- The project currently limits track processing to 10 tracks per run
- HTML content is truncated to 10,000 characters before sending to OpenAI
- Rate limiting is implemented with simple setTimeout delays
- Storage is file-based in the `data/` directory
- No test infrastructure exists yet; don't add tests unless specifically requested

## When Making Changes
1. Follow ESM module syntax with `.js` extensions in imports
2. Maintain TypeScript strict mode compliance
3. Use the established service layer pattern
4. Add JSDoc comments for new exported functions
5. Handle errors gracefully with try-catch and logging
6. Test API endpoints manually after changes
7. Keep changes minimal and focused
8. Don't add testing infrastructure unless requested