Incident Response Runbooks - Deal Scout

Quick Navigation:

High Error Rate
Database Unreachable
Redis Unreachable
Task Queue Stuck
Memory Issues
Disk Full
API Latency
Marketplace Integration Failures
Notification Delivery Failures
Rollback Procedure

High Error Rate

Alert: Error rate > 5% for 5 minutes Severity: CRITICAL Expected Duration: 15-30 minutes

Immediate Actions (0-5 minutes)

Check Dashboard

- Open Grafana dashboard
- Look at error rate trend (last 1 hour)
- Check which endpoints are failing

Verify Service Health

curl -s http://localhost:8000/health | jq .
# Check: db, redis, queue_depth

Check Recent Logs

docker-compose logs --tail 100 backend | grep ERROR
# or
tail -100 /var/log/deal-scout/app.log | grep ERROR

Investigation (5-15 minutes)

Identify Error Pattern
- Is it affecting all endpoints or specific ones?
- Is it new errors or existing issue escalating?
- Check if errors correlate with specific time or event

Check External Dependencies

# Test database
psql $DATABASE_URL -c "SELECT 1"

# Test Redis
redis-cli ping

# Test API integrations
curl -s https://api.ebay.com/api/health

Review Code Changes

git log --oneline -10
git diff HEAD~1..HEAD

Resolution

If Database is Down:

See Database Unreachable runbook

If Redis is Down:

See Redis Unreachable runbook

If Specific Endpoint Failing:

# Check application logs for detailed error
docker-compose logs backend | grep "<endpoint>"

# Check if recent deployment caused issue
docker-compose ps  # Check image version

If Transient Issue:

# Restart backend
docker-compose restart backend

# Monitor error rate
watch -n 5 'curl -s http://localhost:8000/metrics | grep "request.*5"'

Escalation

If error rate remains > 5% after 15 minutes:

Page on-call engineer
Consider rollback (see Rollback Procedure)
Failover to previous deployment

Database Unreachable

Alert: PostgreSQL connection failed Severity: CRITICAL Expected Duration: 20-40 minutes

Immediate Actions (0-2 minutes)

Verify Database Status

# Check if container is running
docker-compose ps postgres

# Check logs
docker-compose logs postgres

# Try connection
psql $DATABASE_URL -c "SELECT 1"

Check Connection Details

# Verify DATABASE_URL is correct
echo $DATABASE_URL

# Check network connectivity
docker-compose exec backend curl -v postgres:5432

Stop Application Traffic

# If managed load balancer, drain connections
# Prevent new requests from being accepted

Investigation (2-10 minutes)

Database Disk Space

# Inside postgres container
docker-compose exec postgres du -sh /var/lib/postgresql/data

# Check container logs
docker-compose logs postgres | tail -50

Database Locks

-- If you can connect
SELECT * FROM pg_locks;
SELECT * FROM pg_stat_activity WHERE state != 'idle';

Memory/CPU Issues
```
docker stats postgres
```

Resolution

If Database is Out of Disk Space:

Free up disk space (delete old logs, backups)
Restart PostgreSQL
Monitor recovery

If Database is Locked/Hung:

# Restart database container
docker-compose restart postgres

# Wait for startup
sleep 30

# Verify
docker-compose exec postgres pg_isready

If Connection Pool Exhausted:

# Restart application to reset connection pool
docker-compose restart backend

If Replication Lag (RDS):

# Failover to replica
# Via AWS console or CLI:
aws rds failover-db-cluster --db-cluster-identifier deal-scout

Recovery Checklist

Database accepting connections
Tables accessible and not corrupted
Replication caught up (if applicable)
Application reconnected successfully
Traffic resumed
Error rate returned to normal

Redis Unreachable

Alert: Redis connection failed Severity: HIGH Expected Duration: 10-20 minutes

Immediate Actions (0-2 minutes)

Verify Redis Status

# Check container
docker-compose ps redis

# Try connection
redis-cli ping

# Check logs
docker-compose logs redis

Check Memory
```
redis-cli info memory
```

Investigation (2-5 minutes)

If Out of Memory

# Check queue depth
redis-cli LLEN celery

# Check memory fragmentation
redis-cli info memory | grep fragmentation

If Corrupted

# Check RDB file
ls -lh /var/lib/redis/dump.rdb

Resolution

Normal Restart:

docker-compose restart redis

# Verify
redis-cli ping

If Out of Memory:

Kill non-critical processes
Reduce max memory
Clear old data

redis-cli FLUSHALL  # WARNING: Clears all data

If Corrupted:

# Backup corrupted file
cp /var/lib/redis/dump.rdb /var/lib/redis/dump.rdb.backup

# Remove corrupted file
rm /var/lib/redis/dump.rdb

# Restart
docker-compose restart redis

Important Notes

Restarting Redis will lose all cached data
Task queue will be cleared (rescan will happen)
Cache warmup will happen gradually
Monitor for spike in database load

Task Queue Stuck

Alert: Queue depth > 1000 for 10 minutes Severity: HIGH Expected Duration: 15-30 minutes

Immediate Actions (0-5 minutes)

Check Queue Status

# Queue depth
redis-cli LLEN celery

# Active tasks
celery -A app.worker inspect active

# Worker stats
celery -A app.worker inspect stats

Check Worker Health

# See if workers are running
docker-compose ps worker

# Check logs
docker-compose logs worker | tail -50

Verify Beat Scheduler

docker-compose ps beat
docker-compose logs beat | tail -50

Investigation (5-15 minutes)

Identify Stuck Tasks

# Get active task details
celery -A app.worker inspect active_queues

# Get task details
celery -A app.worker inspect query_task scan_all

Check Resource Usage

docker stats worker
# Is it CPU bound, memory bound, or waiting?

Check for Errors

# Failed tasks
celery -A app.worker inspect failed

# Reserved tasks
celery -A app.worker inspect reserved

Resolution

If Workers are Dead:

docker-compose restart worker

# Monitor queue clearing
watch -n 5 'redis-cli LLEN celery'

If Workers are Hung:

# Kill hung processes
docker-compose kill worker
docker-compose rm worker
docker-compose up -d worker

If Specific Task Stuck:

# Purge failed tasks
celery -A app.worker purge

# WARNING: This will clear all tasks!
# Use only if workers are completely stuck

If Beat Scheduler Stuck:

docker-compose restart beat

# Verify schedule restored
celery -A app.worker inspect scheduled

Recovery Checklist

Memory Issues

Alert: Memory usage > 85% for 10 minutes Severity: WARNING Expected Duration: 10-20 minutes

Immediate Actions (0-5 minutes)

Check Memory Usage

docker stats
# Identify which service is using memory

# Check inside container
docker-compose exec backend free -h

Check for Memory Leaks

# Monitor growth over time
watch -n 5 'docker stats --no-stream'

Investigation (5-10 minutes)

For Backend:

# Check for large query results
docker-compose logs backend | grep "loaded X objects"

# Check for open connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity"

For Worker:

# Check for stuck tasks
celery -A app.worker inspect active | wc -l

For Database:

# Check buffer usage
psql $DATABASE_URL -c "SELECT * FROM pg_stat_database WHERE datname = 'deals'"

Resolution

Temporary (Quick Relief):

Restart the high-memory service
This may cause brief downtime but releases memory

Permanent (Root Cause):

Identify memory leak source
Deploy fix in next release
Monitor for recurrence

Restart Service:

docker-compose restart backend  # or worker, postgres, redis

# Monitor
docker stats --no-stream

Disk Full

Alert: Disk space < 5% Severity: CRITICAL Expected Duration: 30-60 minutes

Immediate Actions (0-2 minutes)

Check Disk Usage
```
df -h
du -sh /*
```

Identify Large Files

# Find files > 100MB
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null

Investigation (2-10 minutes)

Check Log Files

du -sh /var/log/*
ls -lhS /var/log/deal-scout/

Check Database

du -sh /var/lib/postgresql/data
du -sh /var/backups/postgresql

Check Container Logs

# Docker stores logs
du -sh /var/lib/docker/containers/

Resolution

Clean Log Files:

# Safely delete old logs
find /var/log/deal-scout -name "*.log.*" -mtime +7 -delete

# Or rotate
logrotate -f /etc/logrotate.conf

Clean Database Backups:

# Delete old backups older than 7 days
find /var/backups/postgresql -mtime +7 -delete

Clean Docker Logs:

# Prune unused images/containers/volumes
docker system prune -a

# Clear log files (be careful!)
find /var/lib/docker/containers -name "*.log" -delete

Temporary: Clean tmp Files:

# Remove temp files
rm -rf /tmp/*
rm -rf /var/tmp/*

Prevention

Set up log rotation in /etc/logrotate.d/deal-scout:

/var/log/deal-scout/*.log {
    daily
    rotate 7
    compress
    delaycompress
    notifempty
    create 0644 root root
    postrotate
        docker-compose kill -s HUP backend
    endscript
}

API Latency

Alert: p95 latency > 500ms Severity: WARNING Expected Duration: 10-30 minutes

Investigation

Check Database Queries

# Find slow queries
psql $DATABASE_URL -c "
SELECT mean_exec_time, calls, query
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10"

Check Load

# Current request rate
curl -s http://localhost:8000/metrics | grep "requests_total"

# Connection count
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity"

Check Resources
```
docker stats backend
```

Resolution

Scale Horizontally:

# Increase backend replicas
docker-compose up -d --scale backend=5

Optimize Queries:

# Add indexes (see optimize_database.sql)
psql $DATABASE_URL < scripts/optimize_database.sql

Clear Cache:

redis-cli FLUSHALL
# This will flush all cache, triggering refresh

Restart Backend:

docker-compose restart backend

Marketplace Integration Failures

Alert: eBay API errors > 10 in 5 minutes Severity: WARNING Expected Duration: 15-45 minutes

Investigation

Check API Status
- Visit: https://status.ebay.com
- Check Twitter: @eBayAPIPlatform

Check Credentials

# Verify credentials are still valid
echo $EBAY_APP_ID
echo $EBAY_ENV

Check Recent Errors

docker-compose logs backend | grep -i ebay | tail -20

Resolution

If eBay API is Down:

Wait for restoration
Monitor /health endpoint
Resume scanning once API is up

If Credentials Invalid:

Refresh OAuth token
Update credentials
Restart scanner

If Rate Limited:

Reduce scan frequency
Implement exponential backoff
Check quota usage

Notification Delivery Failures

Alert: Email failure rate > 10% Severity: WARNING Expected Duration: 15-30 minutes

Investigation

Check SMTP Connectivity
```
telnet $SMTP_HOST $SMTP_PORT
```

Check Credentials

echo $SMTP_USER
echo $SMTP_PASSWORD  # Don't share!

Check Recent Errors

docker-compose logs backend | grep -i "smtp\|email" | tail -20

Check Email Queue

# Unsent notifications
psql $DATABASE_URL -c "SELECT count(*) FROM notifications WHERE status = 'pending'"

Resolution

If SMTP Server Down:

Contact email provider
Monitor status page
Retry notifications once restored

If Credentials Invalid:

Update SMTP credentials
Test connection
Restart application

If Rate Limited:

Reduce sending frequency
Batch notifications
Check provider limits

Retry Failed Notifications:

# Manually trigger retry
docker-compose exec backend celery -A app.worker.celery_app -c 1 call app.tasks.notify.send_notifications

Rollback Procedure

Use when: Critical issue requires immediate reversal Duration: 10-15 minutes total

Pre-Rollback

Verify Issue is from Recent Deployment
```
git log --oneline -5
```

Identify Previous Good Version

git log --oneline | grep -i "stable\|release"

Rollback Steps

Stop Current Deployment
```
docker-compose down
```

Checkout Previous Version

git log --oneline -10
git checkout <previous-commit>  # Use commit hash before problematic change

Rebuild Images
```
docker-compose build --no-cache
```
Start Services
```
docker-compose up -d
```

Verify Rollback

# Check health
curl http://localhost:8000/health

# Check logs
docker-compose logs backend | tail -20

# Verify error rate
sleep 30
curl http://localhost:8000/metrics | grep "requests_total"

Post-Rollback

Notify Team
- Slack/Email: "Rollback completed. System stable."
- Document: What was rolled back and why
Root Cause Analysis
- When the incident is stable
- Review what went wrong
- Implement preventive measures

Revert to Good Version

# Pull latest stable
git checkout main
git pull origin main

Escalation Policy

Response Times

Severity	Initial Response	Resolution Target
CRITICAL	< 2 minutes	< 30 minutes
HIGH	< 5 minutes	< 1 hour
WARNING	< 15 minutes	< 4 hours

Escalation Path

On-Call Engineer
- First responder
- Runs runbooks
- Gathers initial data
Senior Engineer
- If not resolved in 15 minutes (CRITICAL) / 30 minutes (HIGH)
- Code review authority
- Database expert
Engineering Lead
- If not resolved in 30 minutes (CRITICAL)
- Deployment authority
- Architecture decisions
Executive/Customer
- If SLA breached or multi-hour outage
- CEO/VP notification
- Customer communication

Contact Information

On-Call: See PagerDuty rotation
Slack: #deal-scout-incidents
Email: team@example.com