System Architecture

Overview

This is a serverless video automation system built on AWS that generates AI video content and publishes it to YouTube on a schedule.

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                        EventBridge Scheduler                     │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────────────┐   │
│  │ Idea Gen     │  │ Video Check  │  │ Upload Schedule    │   │
│  │ (Daily 8am)  │  │ (Every 15min)│  │ (9am,1pm,5pm,9pm)  │   │
│  └──────┬───────┘  └──────┬───────┘  └─────────┬──────────┘   │
└─────────┼──────────────────┼────────────────────┼──────────────┘
          │                  │                    │
          ▼                  ▼                    ▼
    ┌──────────────────────────────────────────────────┐
    │           Orchestrator Lambda                     │
    │  (Coordinates all services)                       │
    └────┬─────────────────┬─────────────────┬─────────┘
         │                 │                 │
         ▼                 ▼                 ▼
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Idea Generator  │ │   Video      │ │    YouTube      │
│    Lambda       │ │  Generator   │ │    Uploader     │
│                 │ │   Lambda     │ │     Lambda      │
│  - GPT-4 API    │ │              │ │                 │
│  - Creates      │ │ - HeyGen/DID │ │ - Downloads     │
│    scripts      │ │ - Async gen  │ │   video         │
│                 │ │ - Status     │ │ - Uploads to    │
│                 │ │   tracking   │ │   YouTube       │
└────────┬────────┘ └──────┬───────┘ └────────┬────────┘
         │                 │                  │
         │                 │                  │
         └─────────────────┼──────────────────┘
                          │
                          ▼
                ┌──────────────────┐
                │    DynamoDB      │
                │  ┌────────────┐  │
                │  │   Ideas    │  │
                │  │   Videos   │  │
                │  │  Uploads   │  │
                │  │   Config   │  │
                │  └────────────┘  │
                └──────────────────┘

┌──────────────────────────────────────────────┐
│         API Gateway (HTTP API)                │
│                                               │
│  GET  /health                                 │
│  GET  /status                                 │
│  GET  /config/topic                           │
│  POST /config/topic                           │
│  POST /trigger/ideas                          │
│  POST /trigger/videos                         │
│  POST /trigger/upload                         │
└───────────────┬──────────────────────────────┘
                │
                ▼
          ┌────────────┐
          │ API Lambda │
          └────────────┘

Data Models

Video Idea

{
  "id": "uuid",
  "topic": "artificial intelligence",
  "title": "AI Will Change Everything in 2024",
  "script": "Hey everyone! Today I want to talk about...",
  "description": "Discover how AI is transforming...",
  "tags": ["ai", "technology", "future"],
  "status": "pending|generating_video|video_ready|uploaded|failed",
  "created_at": "2024-01-10T12:00:00Z",
  "updated_at": "2024-01-10T12:00:00Z",
  "retry_count": 0
}

Generated Video

{
  "id": "uuid",
  "idea_id": "uuid",
  "video_url": "https://...",
  "video_provider": "heygen",
  "provider_video_id": "abc123",
  "duration_seconds": 60,
  "status": "pending|in_progress|completed|failed",
  "created_at": "2024-01-10T12:00:00Z",
  "updated_at": "2024-01-10T12:05:00Z",
  "retry_count": 0,
  "error_message": null
}

YouTube Upload

{
  "id": "uuid",
  "video_id": "uuid",
  "idea_id": "uuid",
  "youtube_video_id": "dQw4w9WgXcQ",
  "youtube_url": "https://youtube.com/watch?v=...",
  "status": "pending|uploading|published|failed",
  "scheduled_time": "2024-01-10T13:00:00Z",
  "uploaded_at": "2024-01-10T13:02:34Z",
  "created_at": "2024-01-10T12:00:00Z",
  "updated_at": "2024-01-10T13:02:34Z",
  "retry_count": 0,
  "error_message": null
}

Workflow Details

1. Idea Generation Flow

EventBridge (8am daily)
  → Orchestrator Lambda
    → Idea Generator Lambda
      → OpenAI GPT-4 API
        ← Returns 8 video ideas with scripts
      → Save to DynamoDB (ideas table)
        - Status: "pending"

2. Video Generation Flow

EventBridge (every 2 hours)
  → Orchestrator Lambda
    → Video Generator Lambda
      → Query DynamoDB for pending ideas
      → For each idea:
        → Call HeyGen/D-ID API
          ← Returns video job ID
        → Update DynamoDB
          - Video status: "in_progress"
          - Idea status: "generating_video"

Separately (every 15 minutes):
EventBridge
  → Orchestrator Lambda
    → Video Generator Lambda (check_status=true)
      → Query videos with status "in_progress"
      → For each video:
        → Check status with provider API
        → If completed:
          - Download video URL
          - Update status: "completed"
          - Update idea status: "video_ready"

3. YouTube Upload Flow

EventBridge (at scheduled times: 9am, 1pm, 5pm, 9pm)
  → Orchestrator Lambda
    → YouTube Uploader Lambda
      → Query DynamoDB for completed videos
      → For each video (limit 1 per trigger):
        → Download video from provider URL
        → Upload to YouTube API
          - Set title, description, tags
          - Set privacy: public
        → Update DynamoDB
          - Upload status: "published"
          - Idea status: "uploaded"
        → Delete temporary file

Error Handling

Retry Strategy

• Max Retries: 3 (configurable)
• Retry Tracking: retry_count field in each table
• Exponential Backoff: Implemented in scheduled checks

Error States

1. Idea Generation Failure

   • Logged to CloudWatch
   • Status: "failed"
   • Does not block other ideas

2. Video Generation Failure

   • Retry up to max_retries
   • If provider fails, status: "failed"
   • Idea status: "failed"
   • Error message stored

3. Upload Failure

   • Retry up to max_retries
   • Common issues:
     • Download failure
     • YouTube API quota exceeded
     • OAuth token expired
   • Error message stored

Scalability

Current Limits

• Ideas per day: 8 (1 batch)
• Videos per day: ~4-8 (depends on generation time)
• Uploads per day: 4 (scheduled times)

Scaling Considerations

1. Increase Frequency

   • Modify EventBridge schedules
   • More posting times

2. Batch Size

   • Increase ideas per batch
   • Process more videos per trigger

3. Concurrent Processing

   • Lambda can process multiple videos simultaneously
   • DynamoDB auto-scales

4. Cost Optimization

   • Use reserved capacity for DynamoDB if traffic is predictable
   • Batch video generation to reduce API calls

Security

IAM Permissions

• Lambda Role: Minimal permissions
  • DynamoDB: Read/Write to specific tables
  • Lambda: Invoke other functions
  • Logs: Write to CloudWatch

Secrets Management

• API keys stored in Lambda environment variables
• Terraform variables marked as sensitive
• Never logged or exposed in responses

YouTube OAuth

• Refresh token stored securely
• Scope limited to upload only
• Token rotation supported

Monitoring

CloudWatch Metrics

1. Lambda Invocations

   • Success/failure rates
   • Duration
   • Concurrent executions

2. DynamoDB

   • Read/write capacity
   • Item counts
   • Throttling

3. API Gateway

   • Request count
   • Latency
   • Error rates

Logging Strategy

All logs use structured JSON format:

{
  "timestamp": "2024-01-10T12:00:00Z",
  "level": "INFO",
  "message": "Video generation started",
  "video_id": "...",
  "provider_video_id": "..."
}

Alarms (Recommended)

1. Lambda errors > 5 in 5 minutes
2. DynamoDB throttling
3. API Gateway 5xx errors
4. Upload failures

Cost Breakdown

AWS Infrastructure

Service	Usage	Monthly Cost
Lambda	~1000 invocations, 10 GB-seconds	$0.50
DynamoDB	On-demand, ~1000 operations/day	$2.00
API Gateway	~100 requests/day	$0.10
CloudWatch	Logs retention 14 days	$1.00
EventBridge	~50 rules invocations/day	$0.00
Total		~$5/month

External APIs (4 videos/day)

Service	Cost per Video	Monthly (120 videos)
OpenAI GPT-4	$0.03-0.06	$3.60-7.20
HeyGen	$0.50-2.00	$60-240
D-ID	$0.30-1.00	$36-120
YouTube	Free	$0

Total Monthly: $65-365 depending on provider and plan

Deployment

Prerequisites

1. AWS Account with CLI configured
2. Terraform >= 1.0
3. API keys obtained
4. YouTube OAuth completed

Deployment Steps

# 1. Install dependencies
pip install -r requirements.txt

# 2. Get YouTube token
python scripts/get_youtube_token.py

# 3. Configure variables
cp terraform/terraform.tfvars.example terraform/terraform.tfvars
# Edit with your values

# 4. Deploy
./scripts/deploy.sh

# 5. Test
./scripts/test_system.sh

Infrastructure Components

• 4 DynamoDB Tables: Ideas, Videos, Uploads, Config
• 5 Lambda Functions: Idea Gen, Video Gen, Upload, Orchestrator, API
• 1 API Gateway: HTTP API with 7 routes
• 7+ EventBridge Rules: Scheduling automation
• 5 CloudWatch Log Groups: Per-function logging
• 1 IAM Role: Lambda execution role with minimal permissions

Maintenance

Regular Tasks

1. Monitor Logs (weekly)

   • Check for repeated errors
   • Review success rates

2. Review Costs (monthly)

   • DynamoDB usage
   • Lambda duration
   • API provider costs

3. Update Topics (as needed)

   • Via API: POST /config/topic
   • Keep content fresh

4. Rotate Credentials (quarterly)

   • YouTube OAuth token
   • API keys
   • Update Terraform variables

Cleanup Old Data

# Delete old ideas (optional)
aws dynamodb scan --table-name video-automation-ideas \
  --filter-expression "created_at < :date" | ...

# Archive uploads to S3
# ... custom script

Future Enhancements

Potential Features

1. Multi-language Support

   • Generate videos in different languages
   • Use different avatars per language

2. A/B Testing

   • Multiple video variants
   • Track performance metrics

3. Content Calendar

   • Schedule specific topics for specific dates
   • Holiday/event-based content

4. Analytics Integration

   • Pull YouTube analytics
   • Optimize based on performance

5. Multiple YouTube Channels

   • Support posting to different channels
   • Different topics per channel

6. Video Templates

   • Predefined formats
   • Consistent branding

7. Thumbnail Generation

   • AI-generated thumbnails
   • A/B test thumbnails

Troubleshooting Guide

Common Issues

1. "YouTube OAuth Error"

   • Regenerate refresh token
   • Verify scopes include youtube.upload

2. "Video Generation Timeout"

   • Increase Lambda timeout
   • Check provider API status

3. "DynamoDB Throttling"

   • Switch to provisioned capacity
   • Reduce concurrent operations

4. "No Videos Being Generated"

   • Check if ideas exist with status "pending"
   • Review video generator logs
   • Verify API key is valid

5. "Uploads Failing"

   • Check YouTube API quota
   • Verify video file is valid
   • Test OAuth credentials

For more help, see README.md or check CloudWatch logs.