DEV Community

How to Build an AI-Powered Portfolio Video Generator: Architecture, Pipelines, and Production Lessons

Building an AI Video Application

Building an AI video application looks deceptively simple from the outside. A user uploads a résumé, portfolio, presentation, or business document. The application extracts the content, generates a script, creates visuals, synthesizes speech, animates a digital presenter, and exports a polished video. That sounds like a sequence of API calls. In production, it is actually a distributed media-processing system involving document parsing, large language models, image generation, text-to-speech, video generation, object storage, background jobs, FFmpeg, retries, progress tracking, security, and cost control.

This guide explains how to design such a system from the ground up. The focus is not on a specific AI provider. Instead, we will create a provider-independent architecture that can support multiple language models, voice engines, image generators, avatar systems, and video-generation APIs.

By the end of this guide, you will understand:

  • How to convert unstructured documents into video scenes
  • How to design a reliable asynchronous generation pipeline
  • How to structure prompts for consistent scripts and visuals
  • How to merge generated media using FFmpeg
  • How to track progress in the frontend
  • How to handle failures, retries, storage, security, and scaling
  • How to avoid common mistakes that make AI video applications expensive or unstable

An AI Video Generator Is a Workflow Engine

The first architectural mistake developers make is treating video generation as a single API request. It is better to think of the system as a workflow engine. A typical video portfolio pipeline might contain the following stages:

Document Upload
↓
Text Extraction
↓
Document Classification
↓
Information Structuring
↓
Script Generation
↓
Scene Planning
↓
Visual Prompt Generation
↓
Image or Background Generation
↓
Voice Generation
↓
Avatar or Motion Generation
↓
Scene Rendering
↓
Video Composition
↓
Final Encoding
↓
Publishing

Every stage can fail independently. For example:

  • PDF parsing may fail because the file contains scanned images.
  • The language model may return invalid JSON.
  • An image API may reject a prompt.
  • A voice provider may time out.
  • A generated video may have a different duration than expected.
  • FFmpeg may run out of memory.
  • Uploading the completed file may fail after rendering succeeds.

Because of this, the entire pipeline should be modeled as a state machine rather than one long synchronous function.

Defining the Generation State Machine

A video project should have a clear status field.

export type ProjectStatus =
  | "created"
  | "uploading"
  | "extracting_content"
  | "generating_script"
  | "planning_scenes"
  | "generating_assets"
  | "generating_audio"
  | "generating_video"
  | "composing"
  | "uploading_output"
  | "completed"
  | "failed";

You may also want statuses at the scene level.

export type SceneStatus =
  | "pending"
  | "generating_image"
  | "generating_audio"
  | "generating_motion"
  | "ready"
  | "failed";

This gives the system several advantages. First, the frontend can show meaningful progress instead of displaying an indefinite spinner. Second, failed jobs can resume from the last successful stage. Third, operators can identify which provider or processing step is creating failures. Fourth, expensive assets do not need to be regenerated unnecessarily.

A simplified project model might look like this:

interface VideoProject {
  id: string;
  userId: string;
  title: string;
  sourceDocumentUrl: string;
  sourceFileType: "pdf" | "docx" | "pptx" | "txt";
  status: ProjectStatus;
  progress: number;
  currentStep?: string;
  scenes: VideoScene[];
  outputVideoUrl?: string;
  errorMessage?: string;
  createdAt: Date;
  updatedAt: Date;
}

Each video scene can be stored independently.

interface VideoScene {
  id: string;
  projectId: string;
  order: number;
  heading: string;
  narration: string;
  visualPrompt: string;
  imageUrl?: string;
  audioUrl?: string;
  motionVideoUrl?: string;
  durationSeconds?: number;
  status: SceneStatus;
}

This data model allows individual scenes to be regenerated without restarting the complete video.

Step 1: Accepting and Validating Documents

The input document is the foundation of the generated video. Applications commonly accept:

  • PDF résumés
  • DOCX files
  • Presentation decks
  • Business profiles
  • Case studies
  • Plain text
  • Markdown
  • Portfolio descriptions

Never trust the filename or MIME type sent by the browser. Validate:

  • File size
  • Actual file signature
  • Allowed extension
  • MIME type
  • Page count
  • Whether the document is encrypted
  • Whether extraction returns meaningful text

A basic Next.js upload endpoint might begin like this:

import { NextRequest, NextResponse } from "next/server";

const MAX_FILE_SIZE = 10 * 1024 * 1024;
const ALLOWED_TYPES = new Set([
  "application/pdf",
  "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
  "text/plain",
]);

export async function POST(request: NextRequest) {
  const formData = await request.formData();
  const file = formData.get("file");

  if (!(file instanceof File)) {
    return NextResponse.json({ error: "A valid file is required" }, { status: 400 });
  }

  if (!ALLOWED_TYPES.has(file.type)) {
    return NextResponse.json({ error: "Unsupported file type" }, { status: 415 });
  }

  if (file.size > MAX_FILE_SIZE) {
    return NextResponse.json({ error: "File exceeds the 10 MB limit" }, { status: 413 });
  }

  const bytes = Buffer.from(await file.arrayBuffer());
  // Verify the real file signature before saving.
  // Upload the original file to object storage.
  // Create a project record.
  // Enqueue the extraction job.

  return NextResponse.json({ status: "accepted" });
}

For production applications, direct browser-to-object-storage uploads are usually better than routing large files through the web application server. The recommended flow is:

Browser requests signed upload URL
↓
Backend creates signed URL
↓
Browser uploads directly to object storage
↓
Browser confirms completed upload
↓
Backend creates processing job

This reduces memory usage and prevents large uploads from consuming application server capacity.

Step 2: Extracting Structured Content

A résumé is not simply a block of text. It contains sections and relationships:

  • Name
  • Professional title
  • Summary
  • Skills
  • Work experience
  • Projects
  • Achievements
  • Education
  • Certifications
  • Contact information

The extraction stage should therefore have two parts. The first part converts the document into raw text. The second part converts that raw text into a normalized schema.

For a résumé, the normalized format might be:

interface ResumeData {
  fullName?: string;
  headline?: string;
  summary?: string;
  skills: string[];
  experience: Array<{
    company?: string;
    role?: string;
    startDate?: string;
    endDate?: string;
    achievements: string[];
  }>;
  projects: Array<{
    name?: string;
    description?: string;
    technologies: string[];
    outcomes: string[];
  }>;
  education: Array<{
    institution?: string;
    qualification?: string;
    year?: string;
  }>;
}

Do not ask a language model to directly generate the final video script from raw extracted text. That creates several problems:

  • Important information may be skipped.
  • Contact details may accidentally appear in narration.
  • Dates may be hallucinated.
  • The script may overemphasize one section.
  • Repeated résumé text may create repetitive narration.

Instead, use a structured intermediate representation. A language model prompt could request strict JSON:

You are extracting structured professional information from a document. Return valid JSON only. Do not invent missing details. Do not infer dates, company names, metrics, or technologies. Ignore headers, footers, page numbers, and duplicated content.

Schema:
{
  "fullName": "string or null",
  "headline": "string or null",
  "summary": "string or null",
  "skills": ["string"],
  "experience": [
    {
      "company": "string or null",
      "role": "string or null",
      "startDate": "string or null",
      "endDate": "string or null",
      "achievements": ["string"]
    }
  ],
  "projects": [
    {
      "name": "string or null",
      "description": "string or null",
      "technologies": ["string"],
      "outcomes": ["string"]
    }
  ]
}

The response should be validated against a schema using a library such as Zod.

import { z } from "zod";

const ResumeSchema = z.object({
  fullName: z.string().nullable(),
  headline: z.string().nullable(),
  summary: z.string().nullable(),
  skills: z.array(z.string()),
  experience: z.array(
    z.object({
      company: z.string().nullable(),
      role: z.string().nullable(),
      startDate: z.string().nullable(),
      endDate: z.string().nullable(),
      achievements: z.array(z.string()),
    })
  ),
  projects: z.array(
    z.object({
      name: z.string().nullable(),
      description: z.string().nullable(),
      technologies: z.array(z.string()),
      outcomes: z.array(z.string()),
    })
  ),
});

Never assume that “JSON mode” guarantees valid application data. It may guarantee syntactically valid JSON while still returning missing fields, incorrect data types, duplicated values, or unsupported structures. Always validate and normalize the result.

Step 3: Creating the Narrative Strategy

Once the content is structured, the system needs to decide what kind of story to tell. A good portfolio video is not a résumé being read aloud. Reading every role, skill, date, and certification creates a long and uninteresting video. Instead, convert the source material into a narrative.

A useful structure is:

  • Who the person is
  • What problem they solve
  • What experience supports that claim
  • What work demonstrates their capability
  • What outcomes they have produced
  • What type of opportunity they are seeking
  • How the viewer can continue the conversation

The narrative strategy can change according to the user’s goal. For example, a job-seeking video could emphasize:

  • Role fit
  • Technical skills
  • Relevant achievements
  • Communication ability
  • Career direction

A freelancer portfolio could emphasize:

  • Client problems
  • Services
  • Project examples
  • Measurable outcomes
  • Working style

A founder presentation could emphasize:

  • Problem
  • Market context
  • Product
  • Differentiation
  • Traction
  • Vision

This means “video type” should be a first-class input.

type VideoPurpose =
  | "job_search"
  | "freelance_portfolio"
  | "founder_pitch"
  | "consultant_profile"
  | "personal_brand"
  | "project_case_study";

The selected purpose can control the script prompt, scene count, tone, visual style, and call to action.

Step 4: Planning Scenes Before Generating Assets

Do not generate images, audio, or motion until the scene plan has been approved by your backend validation logic.

A scene plan might look like this:

{
  "videoTitle": "Building Reliable Developer Platforms",
  "targetDurationSeconds": 75,
  "tone": "professional and conversational",
  "scenes": [
    {
      "order": 1,
      "heading": "Introduction",
      "narration": "I build developer platforms that make complex systems easier to operate.",
      "visualConcept": "A modern developer workspace with abstract cloud infrastructure",
      "durationSeconds": 8
    },
    {
      "order": 2,
      "heading": "Core Expertise",
      "narration": "My work focuses on backend architecture, automation, and scalable media pipelines.",
      "visualConcept": "Connected service nodes representing APIs, queues, storage, and compute",
      "durationSeconds": 10
    }
  ]
}

Validate the following rules:

  • Scene count must be within a reasonable limit.
  • Narration must not be empty.
  • Narration length must match the estimated duration.
  • The total duration must remain within the user’s plan allowance.
  • Visual prompts must not contain unsupported content.
  • The final scene should contain a useful conclusion.
  • No private data should appear unless explicitly allowed.

A rough narration estimate is 130 to 160 spoken words per minute. A 10-second scene should usually contain around 22 to 27 words. You can estimate duration with:

function estimateNarrationDuration(narration: string, wordsPerMinute = 145): number {
  const words = narration.trim().split(/\s+/).filter(Boolean).length;
  return Math.ceil((words / wordsPerMinute) * 60);
}

This estimate will not be exact because pauses, punctuation, voice style, and provider settings affect speech duration. After audio generation, replace the estimate with the real audio duration obtained through ffprobe.

Step 5: Designing Provider-Independent Interfaces

AI APIs change quickly. Pricing changes. Models disappear. Rate limits change. Output formats evolve. Providers sometimes experience outages. Your application should not embed provider-specific logic throughout the codebase. Use interfaces.

interface ScriptGenerator {
  generateScenePlan(input: {
    structuredDocument: ResumeData;
    purpose: VideoPurpose;
    targetDurationSeconds: number;
    tone: string;
  }): Promise<ScenePlan>;
}

interface ImageGenerator {
  generateImage(input: {
    prompt: string;
    aspectRatio: "16:9" | "9:16" | "1:1";
  }): Promise<{ url: string; providerJobId?: string }>;
}

interface SpeechGenerator {
  generateSpeech(input: {
    text: string;
    voiceId: string;
  }): Promise<{ audioUrl: string; durationSeconds?: number }>;
}

interface MotionGenerator {
  generateVideo(input: {
    imageUrl: string;
    audioUrl?: string;
    motionPrompt?: string;
  }): Promise<{ videoUrl: string; providerJobId?: string }>;
}

Your orchestration layer should depend on these interfaces rather than a vendor SDK.

class ScenePipeline {
  constructor(
    private readonly imageGenerator: ImageGenerator,
    private readonly speechGenerator: SpeechGenerator,
    private readonly motionGenerator: MotionGenerator
  ) {}

  async process(scene: VideoScene): Promise<VideoScene> {
    const image = await this.imageGenerator.generateImage({
      prompt: scene.visualPrompt,
      aspectRatio: "16:9",
    });
    const speech = await this.speechGenerator.generateSpeech({
      text: scene.narration,

(The article continues with more steps, but the provided text ends here. The reformatting should cover the given content. The final output will be the Markdown body as described.)

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.