Back to .md Directory

AI Video Generator — Comprehensive Implementation Plan

Build a Next.js web application that generates fully composed, AI-powered videos from a single text prompt. The user provides a topic, and the system generates a scene-by-scene video with AI-generated clips, visual assets, background music, text overlays, and captions — exported as a final MP4.

May 2, 2026
0 downloads
1 views
ai llm rag prompt gemini
View source

AI Video Generator — Comprehensive Implementation Plan

1. Project Vision

Build a Next.js web application that generates fully composed, AI-powered videos from a single text prompt. The user provides a topic, and the system generates a scene-by-scene video with AI-generated clips, visual assets, background music, text overlays, and captions — exported as a final MP4.


2. Architecture Overview

Approach A: Remotion + Gemini Ecosystem

User Prompt
    |
    v
[Gemini 2.5 Flash] -- Structured scene-by-scene script (JSON via Zod schema)
    |
    +---> [Nano Banan Pro] -- Keyframe images, title cards, overlays (per scene)
    |
    +---> [Veo 3.1] -- AI video clips (per scene, 4-8 seconds each)
    |
    +---> [Lyria 2 / RealTime] -- Background music / soundtrack
    |
    v
[Asset Storage (S3/GCS/R2)] -- Download & host all generated assets
    |
    v
[Remotion Composition] -- Compose: Veo clips + Nano Banan assets + text overlays + Lyria audio
    |
    +---> [<Player>] -- In-browser preview
    |
    +---> [renderMedia() via Job Queue] -- Server-side MP4 export
    |
    v
Final MP4 delivered to user

Core Technology Stack

LayerTechnologyPurpose
FrameworkNext.js (App Router)UI, API routes, hosting
Video CompositionRemotionCompose clips, overlays, audio; preview & export
Script GenerationGemini 2.5 FlashStructured scene scripts via Zod schemas
Image GenerationNano Banan Pro (gemini-3-pro-image-preview)Keyframes, title cards, transparent overlays
Video GenerationVeo 3.1 (veo-3.1-generate-preview / veo-3.1-fast-generate-preview)AI video clips per scene
Music GenerationLyria 2 (lyria-002 on Vertex AI) / Lyria RealTime (experimental)Background soundtrack
Asset StorageS3 / GCS / Cloudflare R2Store downloaded Veo clips, images, audio
Job QueueBullMQ + RedisOrchestrate async pipeline & render jobs
SDK@google/genaiUnified SDK for Gemini, Veo, Nano Banan Pro

3. Detailed Component Specifications

3.1 Script Generation — Gemini 2.5 Flash

Model: gemini-2.5-flash (GA, free tier available) Pricing: $0.30 input / $2.50 output per 1M tokens (~$0.01 per script)

Structured Output Schema (Zod):

import { z } from "zod";

const SceneSchema = z.object({
  scene_number: z.number(),
  title: z.string().describe("Short scene title"),
  visual_description: z.string().describe("Detailed visual description for Veo video prompt"),
  image_prompt: z.string().describe("Prompt for Nano Banan Pro keyframe/overlay generation"),
  narration_text: z.string().describe("Narration or caption text for this scene"),
  duration_seconds: z.number().min(4).max(8).describe("Scene duration (4, 6, or 8 seconds)"),
  camera_direction: z.string().describe("Camera movement: pan, zoom, static, tracking, etc."),
  mood: z.string().describe("Emotional tone: dramatic, upbeat, calm, tense, etc."),
  transition: z.enum(["cut", "fade", "dissolve", "wipe"]).describe("Transition to next scene"),
});

const ScriptSchema = z.object({
  title: z.string(),
  theme: z.string().describe("Overall theme/genre of the video"),
  target_audience: z.string(),
  music_prompt: z.string().describe("Prompt for Lyria background music generation"),
  scenes: z.array(SceneSchema).min(3).max(8),
  total_duration_seconds: z.number(),
});

API Call Pattern:

import { GoogleGenAI } from "@google/genai";
import { zodToJsonSchema } from "zod-to-json-schema";

const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

const response = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: `Generate a cinematic video script about: ${userPrompt}`,
  config: {
    responseMimeType: "application/json",
    responseJsonSchema: zodToJsonSchema(ScriptSchema),
  },
});

const script = ScriptSchema.parse(JSON.parse(response.text));

Important Notes:

  • Gemini 2.0 models retiring June 1, 2026 — do NOT use them
  • Property ordering in output matches schema key ordering (Gemini 2.5+)
  • Free tier available for development

3.2 Image Generation — Nano Banan Pro

Model: gemini-3-pro-image-preview (or faster variant gemini-3.1-flash-image) Pricing: $0.134/image (1K/2K), $0.24/image (4K)

Capabilities:

  • Text-to-image with industry-leading text rendering
  • Up to 4K resolution
  • Transparent background generation (for overlays)
  • Multi-image blending (up to 14 reference images)
  • Identity preservation across up to 5 subjects

Role in Pipeline:

  • Generate keyframe images per scene (can be used as Veo reference frames)
  • Generate title cards and intro/outro frames with legible text
  • Generate transparent overlay assets (logos, lower-thirds, watermarks)
  • NOT a video model — purely for static visual assets

API Call Pattern:

const response = await ai.models.generateContent({
  model: "gemini-3-pro-image-preview",
  contents: [{ text: scene.image_prompt }],
});
// Extract image from response, upload to asset storage

Limitations:

  • All outputs watermarked (visible + SynthID)
  • Aggressive content filtering (famous IP, safety policies)
  • Consistency drift across multiple generations
  • Fine detail struggles with complex scenes
  • Billing must be enabled (no free tier on Google Cloud)

3.3 Video Generation — Veo 3.1

Models:

ModelIDCostBest For
Veo 3.1 Standardveo-3.1-generate-preview$0.40/secFinal quality
Veo 3.1 Fastveo-3.1-fast-generate-preview$0.15/secDevelopment, iteration
Veo 3veo-3-generate$0.40/secStable fallback

Specifications:

SpecOptions
Duration4, 6, or 8 seconds per clip
Resolution720p (default), 1080p, 4K
Aspect Ratio16:9 (landscape), 9:16 (portrait)
Frame Rate24 fps
AudioNative audio generation (Veo 3+)
ExtensionsUp to 20 extensions (+7 sec each, max ~148 sec total)

Async Generation Workflow:

// 1. Submit generation job
let operation = await ai.models.generateVideos({
  model: "veo-3.1-fast-generate-preview",  // Use fast for dev
  prompt: scene.visual_description,
  config: {
    aspectRatio: "16:9",
    resolution: "720p",
    numberOfVideos: 1,
    personGeneration: "allow_all",
  },
});

// 2. Poll until complete (10-second intervals)
while (!operation.done) {
  await new Promise((resolve) => setTimeout(resolve, 10000));
  operation = await ai.operations.getVideosOperation({ operation });
}

// 3. Download immediately (48-hour retention!)
const videoFile = operation.response.generatedVideos[0].video;
await ai.files.download({
  file: videoFile,
  downloadPath: `./tmp/${scene.scene_number}.mp4`,
});

// 4. Upload to persistent storage (S3/GCS/R2)
const permanentUrl = await uploadToStorage(`./tmp/${scene.scene_number}.mp4`);

Critical Limitations & Gotchas:

  1. 48-hour retention — Videos deleted after 2 days. MUST download immediately and re-host.
  2. 8-second max per clip — Longer scenes need video extension (adds 7s per hop, max 20 extensions)
  3. Latency variance — 11 seconds to 6 minutes per clip. Pipeline bottleneck.
  4. Text rendering is poor — Avoid prompts requiring readable text in video. Use Remotion text overlays instead.
  5. Hand/finger anomalies — Fine motor details look unnatural in close-ups.
  6. Safety filters are aggressive — Words like "fire", "shot", "strike" can trigger rejection. Have prompt sanitization.
  7. Only 1 video per request — Cannot batch.
  8. personGeneration required — Must set to "allow_all" for text-to-video.
  9. No free tier — Every API call is billed.
  10. Video extensions limited to 720p — Cannot extend at higher resolutions.
  11. SynthID watermark — All outputs watermarked.

Cost Estimates:

ScenarioCalculationTotal
5 scenes x 6s (Fast, 720p)30s x $0.15$4.50
5 scenes x 8s (Fast, 720p)40s x $0.15$6.00
5 scenes x 6s (Standard, 720p)30s x $0.40$12.00
5 scenes x 8s (Standard, 1080p)40s x $0.40$16.00

3.4 Music Generation — Lyria

Current API Options (March 2026):

ModelAccessQualityFeaturesPriceStatus
Lyria 3Gemini App onlyBestVocals, lyrics, multilingualFree (consumer)NO API
Lyria 2Vertex AI APIGoodInstrumental only$0.06/30s clipProduction (allowlist)
Lyria RealTimeGemini API v1alphaStreamingReal-time steeringFreeExperimental

Recommended: Lyria 2 on Vertex AI for v1

POST https://{LOCATION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/lyria-002:predict
  • Input: text prompt, optional negative_prompt, optional seed
  • Output: base64-encoded WAV, 30 seconds, 48 kHz, instrumental only
  • Requires allowlist request (not instant self-service)

Lyria RealTime API (Alternative / Experimental):

WebSocket: wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateMusic
Model: models/lyria-realtime-exp
  • Generates continuous streaming music in 2-second chunks at 48kHz stereo
  • Real-time steering: change genre, instruments, mood, key, tempo on the fly
  • Currently free
  • Best for: interactive use, not batch video rendering

30-Second Limit Problem:

  • All Lyria models cap at 30 seconds per generation
  • For videos > 30 seconds: generate multiple clips with matching style prompts
  • Use Remotion's <Audio> crossfading to blend clips at transitions
  • Alternative: use royalty-free music library as fallback

Specifications:

  • Audio Quality: 24-bit, 48 kHz sample rate
  • Duration: 30 seconds max per generation
  • Output: WAV (Lyria 2) or streaming chunks (RealTime)
  • Languages: 8 languages for vocals (Lyria 3 only, no API)

3.5 Video Composition — Remotion

Core Architecture:

remotion/
  Root.tsx                    # registerRoot() entry point
  compositions/
    AIVideo.tsx               # Main composition: assembles scenes
  sequences/
    SceneSequence.tsx          # Single scene: Veo clip + overlay
    TitleSequence.tsx          # Intro title card (Nano Banan image)
    OutroSequence.tsx          # Outro/credits
  components/
    TextOverlay.tsx            # Animated text/caption overlay
    CaptionRenderer.tsx        # Word-by-word captions
    TransitionEffect.tsx       # Scene transitions (fade, dissolve)

Key Remotion APIs:

APIPurpose
<Composition>Register a video with id, component, dimensions, fps, duration
<Sequence> / <Series>Time-shift scenes, arrange clips back-to-back
<OffthreadVideo src={url}>Embed Veo video clips (supports remote URLs)
<Audio src={url}>Embed Lyria music tracks (supports remote URLs)
interpolate()Animate CSS properties (opacity, position, scale)
spring()Natural bounce animations
calculateMetadata()Dynamically compute duration from AI-generated content
<Player>In-browser preview component
renderMedia()Server-side MP4 export

Composition Pattern:

import { AbsoluteFill, Sequence, OffthreadVideo, Audio, useCurrentFrame, interpolate } from "remotion";

const AIVideo: React.FC<{ script: Script }> = ({ script }) => {
  let currentFrame = 0;

  return (
    <AbsoluteFill>
      {/* Background music (full duration) */}
      <Audio src={script.musicUrl} volume={0.3} />

      {/* Scenes arranged sequentially */}
      {script.scenes.map((scene, i) => {
        const durationInFrames = scene.duration_seconds * 30; // 30fps
        const from = currentFrame;
        currentFrame += durationInFrames;

        return (
          <Sequence key={i} from={from} durationInFrames={durationInFrames}>
            {/* Veo video clip */}
            <OffthreadVideo src={scene.videoUrl} />

            {/* Text overlay with animation */}
            <TextOverlay text={scene.narration_text} />
          </Sequence>
        );
      })}
    </AbsoluteFill>
  );
};

Dynamic Duration with calculateMetadata():

<Composition
  id="AIVideo"
  component={AIVideo}
  fps={30}
  width={1920}
  height={1080}
  calculateMetadata={async ({ props }) => {
    const totalSeconds = props.script.scenes.reduce(
      (sum, s) => sum + s.duration_seconds, 0
    );
    return { durationInFrames: totalSeconds * 30 };
  }}
/>

Configuration Defaults:

  • Resolution: 1920x1080 (1080p)
  • Frame Rate: 30 fps
  • Codec: h264 (MP4)
  • Image Format: jpeg (faster than png unless transparency needed)

Licensing:

  • Free for individuals and teams of 3 or fewer
  • Company License (4+ people): $25/month per developer, minimum $100/month

3.6 Asset Storage (REQUIRED)

Problem: Veo deletes generated videos after 48 hours. All assets must be downloaded and re-hosted immediately.

Recommended: Cloudflare R2 (S3-compatible, no egress fees)

Storage Pipeline:

  1. Veo generates clip -> download to temp directory
  2. Upload to R2/S3/GCS -> get permanent public URL
  3. Pass permanent URL to Remotion's <OffthreadVideo src={permanentUrl}>
  4. Same for Nano Banan Pro images and Lyria audio
  5. Clean up temp directory
  6. Set TTL on stored assets (7-30 days) to manage costs

Estimated Storage per Video:

  • 5 Veo clips (6s each, 720p): ~50-100 MB
  • 5 Nano Banan images: ~5-10 MB
  • 1 Lyria audio clip: ~5 MB
  • Final rendered MP4: ~30-60 MB
  • Total: ~100-175 MB per video

3.7 Job Queue & Pipeline Orchestration

Problem: The pipeline is async (Veo polling), CPU-intensive (Remotion rendering), and multi-step. A simple API route cannot handle this.

Recommended: BullMQ + Redis

Job Flow:

[API Route: POST /api/generate]
    |
    v
[Job: generate-script] -- Gemini 2.5 Flash
    |
    v (on completion)
[Jobs: generate-assets] -- Parallel:
    +-- [generate-image-1..N] -- Nano Banan Pro (per scene)
    +-- [generate-video-1..N] -- Veo 3.1 (per scene, with polling)
    +-- [generate-music]      -- Lyria 2
    |
    v (when all assets ready)
[Job: compose-video] -- Remotion renderMedia()
    |
    v
[Job: deliver] -- Notify user, provide download URL

Progress Updates:

  • WebSocket connection from client to server
  • Each job emits progress events (% complete, current step)
  • Client shows real-time progress UI with scene-by-scene status

4. Risk Analysis & Mitigations

4.1 Showstoppers Identified

#RiskSeverityMitigation
1Visual consistency across Veo clips — characters, lighting, color grading vary between scenesCriticalConstrain v1 to abstract/motion-graphics style; use consistent Remotion color grading overlays; explore Veo image-to-video anchoring; add style-locking prompt patterns
2Lyria API uncertainty — no Lyria 3 API, Lyria 2 requires allowlist, RealTime is v1alphaHighDefault to royalty-free music library for v1; Lyria as premium opt-in; monitor Lyria 3 API announcement
348-hour Veo retention — clips deleted if not downloadedHighImmediate download + upload to R2/S3 in pipeline; retry logic; never reference Veo URLs in Remotion
4Rendering infrastructure — renderMedia() too heavy for serverlessHighDedicated render worker (VM with 8GB+ RAM); or Remotion Lambda; never render in Next.js API routes

4.2 Serious Risks

#RiskImpactMitigation
1Cost: $5-13+ per videoFreemium model not viableCharge per video ($2-5 markup); tiered pricing; per-user daily limits; use Veo Fast for drafts
2Latency: 1-10 minutesPoor UX, user abandonmentAsync UX (email/push notification when ready); real-time progress via WebSocket; parallelize all asset generation
3Veo vendor lock-inNo fallback if Veo goes downAbstract video generation behind interface; monitor Runway/Kling as potential fallbacks
4Architecture complexityDevelopment overhead, debugging difficultyPhase the rollout (see Section 6); start with Gemini + Veo + Remotion only

4.3 SynthID Watermarking

All AI-generated content (Nano Banan images, Veo clips) contains SynthID watermarks:

  • Invisible to humans in most cases
  • Detectable by Google's classifier
  • Cannot be reliably removed
  • Commercial impact: Must disclose to users that content is AI-generated. Not a blocker for social media / marketing use cases. May be a concern for broadcast / professional video.

5. Cost Model

Per-Video Cost Breakdown (5 scenes, 6 seconds each, Fast tier)

ComponentCost
Gemini 2.5 Flash (script)$0.01
Nano Banan Pro (5 keyframes)$0.67
Veo 3.1 Fast (30 sec total)$4.50
Lyria 2 (1 music clip)$0.06
R2 Storage (~150 MB, 30 days)$0.002
Compute (render worker)~$0.10
Total API + Infrastructure~$5.34

Suggested Pricing Tiers

TierPriceIncludesMargin
Free Trial$01 video (720p, 3 scenes)Loss leader (~$2.50)
Basic$9.99/mo5 videos (720p)~$24 margin
Pro$29.99/mo20 videos (1080p)~$60 margin
Pay-as-you-go$3.99/video1 video (720p)~$1.50 margin

6. Phased Implementation Roadmap

Phase 1: Core Pipeline (MVP)

Goal: Text prompt -> AI video with clips and text overlays -> MP4 download

Components:

  • Next.js App Router (UI + API routes)
  • Gemini 2.5 Flash (script generation with Zod schema)
  • Veo 3.1 Fast (video clip generation, 720p)
  • Remotion (composition + rendering)
  • Asset storage (R2 or S3)
  • Basic job queue (BullMQ + Redis)

NOT included in Phase 1: Nano Banan Pro, Lyria, advanced overlays

Deliverables:

  • Next.js project setup with Remotion integration
  • Gemini script generation API route with Zod schema
  • Veo video generation with async polling + download pipeline
  • Asset storage upload/retrieval (R2)
  • Remotion composition: Veo clips + basic text overlays
  • Remotion Player preview in browser
  • BullMQ job queue for pipeline orchestration
  • Render worker (dedicated process or Lambda)
  • Basic progress UI (WebSocket)
  • MP4 download endpoint

Phase 2: Visual Assets & Polish

Goal: Add AI-generated images, better overlays, captions, transitions

Components:

  • Nano Banan Pro (keyframe images, title cards, transparent overlays)
  • Enhanced Remotion compositions (animated captions, transitions, intro/outro)
  • Style consistency improvements (prompt engineering, color grading overlays)

Deliverables:

  • Nano Banan Pro integration for keyframe generation
  • Title card / intro sequence using Nano Banan images
  • Animated caption system (word-by-word, TikTok-style)
  • Scene transition effects (fade, dissolve, wipe)
  • Consistent color grading overlay across all scenes
  • Image-to-video anchoring exploration (Veo reference frames)

Phase 3: Audio & Music

Goal: Add AI-generated background music and sound design

Components:

  • Lyria 2 (Vertex AI) or Lyria RealTime for background music
  • Royalty-free music library as fallback
  • Audio mixing in Remotion (music + narration + Veo native audio)

Deliverables:

  • Lyria 2 integration (Vertex AI allowlist request)
  • Music generation from script mood/theme
  • Audio mixing: background music volume ducking under narration
  • Fallback: curated royalty-free music library
  • Handle 30-second limit (crossfade multiple clips for longer videos)
  • Explore Lyria RealTime API for continuous soundtrack

Phase 4: Scale & Production

Goal: Handle concurrent users, cost controls, production hardening

Deliverables:

  • Render queue scaling (multiple workers, auto-scaling)
  • Per-user generation limits and usage tracking
  • Stripe billing integration (subscription + pay-per-video)
  • CDN for delivered videos
  • Error recovery (retry failed scenes, partial regeneration)
  • Monitoring & alerting (API costs, failure rates, latency)
  • Asset cleanup (TTL-based deletion of stored clips)
  • Rate limit handling for all external APIs

7. Project Structure

src/
  app/
    page.tsx                          # Landing page / prompt input
    generate/
      page.tsx                        # Generation progress & preview page
    api/
      generate/
        route.ts                      # POST: Start video generation pipeline
      status/
        [jobId]/
          route.ts                    # GET: Poll job status
      download/
        [videoId]/
          route.ts                    # GET: Download final MP4
  components/
    PromptInput.tsx                    # User prompt input form
    GenerationProgress.tsx            # Real-time progress UI
    VideoPreview.tsx                   # Remotion Player wrapper
    SceneCard.tsx                      # Individual scene preview card
  remotion/
    Root.tsx                          # Remotion registerRoot()
    compositions/
      AIVideo.tsx                     # Main video composition
    sequences/
      SceneSequence.tsx               # Scene: Veo clip + overlay
      TitleSequence.tsx               # Intro title card
      OutroSequence.tsx               # Outro/credits
    components/
      TextOverlay.tsx                 # Animated text overlay
      CaptionRenderer.tsx             # Word-by-word captions
      TransitionEffect.tsx            # Scene transitions
  lib/
    gemini.ts                         # Gemini API client + prompt templates
    veo.ts                            # Veo API client + polling logic
    nano-banan.ts                     # Nano Banan Pro client
    lyria.ts                          # Lyria API client
    storage.ts                        # R2/S3 upload/download
    types.ts                          # Shared TypeScript types (Script, Scene)
    schemas.ts                        # Zod schemas for structured output
  queue/
    worker.ts                         # BullMQ worker process
    jobs/
      generate-script.ts              # Gemini script generation job
      generate-video.ts               # Veo video generation job
      generate-image.ts               # Nano Banan Pro image job
      generate-music.ts               # Lyria music generation job
      compose-video.ts                # Remotion renderMedia job
      deliver.ts                      # Final delivery job
  styles/
    globals.css                       # Global styles / Tailwind
remotion.config.ts                    # Remotion bundler configuration
docker-compose.yml                    # Redis + render worker

8. Dependencies

{
  "dependencies": {
    "next": "^15",
    "react": "^19",
    "react-dom": "^19",
    "remotion": "^4",
    "@remotion/cli": "^4",
    "@remotion/renderer": "^4",
    "@remotion/player": "^4",
    "@remotion/bundler": "^4",
    "@google/genai": "latest",
    "zod": "^3",
    "zod-to-json-schema": "^3",
    "bullmq": "^5",
    "ioredis": "^5",
    "@aws-sdk/client-s3": "^3",
    "tailwindcss": "^4"
  }
}

9. Environment Variables

# Google AI
GEMINI_API_KEY=                       # Google AI Studio API key (Gemini + Veo + Nano Banan)

# Vertex AI (for Lyria 2)
GOOGLE_CLOUD_PROJECT_ID=
GOOGLE_CLOUD_LOCATION=us-central1
GOOGLE_APPLICATION_CREDENTIALS=       # Service account JSON path

# Asset Storage (R2 example)
R2_ACCOUNT_ID=
R2_ACCESS_KEY_ID=
R2_SECRET_ACCESS_KEY=
R2_BUCKET_NAME=
R2_PUBLIC_URL=

# Redis (for BullMQ)
REDIS_URL=redis://localhost:6379

# App
NEXT_PUBLIC_APP_URL=http://localhost:3000

10. Key API Reference

APISDKModel IDEndpoint
Gemini 2.5 Flash@google/genaigemini-2.5-flashai.models.generateContent()
Nano Banan Pro@google/genaigemini-3-pro-image-previewai.models.generateContent()
Nano Banan 2 (faster)@google/genaigemini-3.1-flash-imageai.models.generateContent()
Veo 3.1 Standard@google/genaiveo-3.1-generate-previewai.models.generateVideos()
Veo 3.1 Fast@google/genaiveo-3.1-fast-generate-previewai.models.generateVideos()
Lyria 2Vertex AI RESTlyria-002Vertex AI predict endpoint
Lyria RealTime@google/genailyria-realtime-expWebSocket streaming

11. Devil's Advocate Summary

What Could Go Wrong

  1. Visual inconsistency is the #1 product risk. Each Veo clip is generated independently — characters, lighting, and style will vary across scenes. No proven solution exists today. Mitigation: constrain to motion-graphics/abstract style for v1, or invest in prompt engineering for style-locking.

  2. $5-13 per video makes free tiers impossible. Must charge from day one or eat costs. This is a premium product, not a mass-market tool.

  3. 1-10 minute generation time. Users will not wait on a loading spinner. Must design for async delivery (email/push notification when ready) with real-time progress updates.

  4. Veo is a single point of failure. No alternative AI video API at comparable quality/price. If Veo goes down, pricing changes, or rate limits tighten, the product is blocked.

  5. Lyria is not production-ready for this use case. Lyria 3 has no API. Lyria 2 needs allowlist approval and is instrumental-only. RealTime is experimental. Default to royalty-free music.

  6. renderMedia() cannot run in serverless. Need a dedicated render worker or Remotion Lambda. This adds infrastructure complexity.

  7. 7 external dependencies for the full pipeline. Start with 3 (Gemini + Veo + Remotion) and add incrementally.

What's Surprisingly Strong

  1. Single SDK (@google/genai) covers Gemini, Veo, and Nano Banan Pro. Clean integration.
  2. Remotion's AI support is excellent — official LLM prompts, agent skills, calculateMetadata for dynamic content.
  3. Veo 3.1 Fast at $0.15/sec is competitively priced for AI video.
  4. Gemini structured output with Zod is reliable and well-documented.
  5. The phased approach (MVP with 3 deps, then add) manages complexity well.

12. Success Criteria

MVP (Phase 1)

  • User can enter a text prompt and receive a composed MP4 video
  • Video contains 3-5 AI-generated scenes with text overlays
  • Generation completes within 5 minutes (p90)
  • Real-time progress shown to user
  • Video preview available before download
  • All assets stored persistently (not reliant on Veo 48h retention)

Production (Phase 4)

  • Supports 50+ concurrent users
  • Per-user billing and usage limits
  • Average cost per video under $6
  • Generation failure rate under 5%
  • Automated retry for failed scenes
  • Music integration (Lyria or royalty-free fallback)

Related Documents