Local Video Summarization Pipeline: Processing Frames with SmolVLM2-2.2B

Local Video Summarization Pipeline: Processing Frames with SmolVLM2-2.2B

SmolVLM2-2.2B sits at a genuinely useful point on the capability-size trade-off curve; small enough to run on a single consumer GPU, capable enough to produce video summaries that are actually useful for real workflows.

Introduction

Most video understanding tools fall into one of two camps. The first camp requires a cloud API; your footage is uploaded, processed on someone else's servers, and billed per minute of video. The second camp runs locally but demands the kind of GPU cluster most developers do not have: 70B+ models that need multiple A100s and take minutes per clip. Neither option works for a developer who wants to process a day's worth of meeting recordings, a lecture series, or security footage on a workstation they already own.

SmolVLM2-2.2B-Instruct, released by Hugging Face on February 20, 2025, changes the calculation. It runs on 5.2 GB of GPU RAM, an RTX 3060, a MacBook Pro M2, and the free Google Colab T4 tier. On Video-MME, the standard long-form video understanding benchmark, it outperforms every existing 2B-scale model. That combination, consumer hardware paired with results that actually hold up, is what this article is built around.

The project we will build in this article: a local pipeline that takes any video file, extracts frames at configurable intervals, analyzes them in batches with SmolVLM2-2.2B, and outputs a structured JSON summary, including per-frame scene descriptions, key moments with timestamps, action items, and a final narrative. The same pipeline handles meeting recordings, lectures, and surveillance footage without changing a line of code.

SmolVLM2-2.2B

The reason SmolVLM2-2.2B can run on an RTX 3060 while outperforming larger models on video tasks is a design decision about how images are tokenized. Most vision-language models tokenize images at high density. Qwen2-VL, for example, uses up to 16,000 tokens to represent a single image. Feeding 50 frames to such a model at that density would consume 800,000 tokens, far beyond any consumer GPU's context budget.

SmolVLM2 uses a pixel shuffle strategy that compresses each 384x384 image patch to 81 tokens. Fifty frames become approximately 4,050 image tokens, manageable in a single inference call. That compression is why SmolVLM2's prefill throughput runs 3.3 to 4.5 times faster and generation throughput runs 7.5 to 16 times faster than Qwen2-VL-2B, not as a marketing claim but as a direct consequence of the token budget difference.

The model comes in three sizes. The 256M and 500M variants are designed for mobile and edge devices; the 256M can run on a phone. The 2.2B is the right choice for this pipeline. It is the only size with strong enough video benchmark scores to produce reliable multi-scene summaries:

• Video-MME of 52.1
• MLVU of 55.2
• MVBench of 46.27

Against the 500M's 42.2, 47.3, and 39.73, respectively.

The video understanding approach is also worth understanding before you write any code. SmolVLM2 does not have a native video encoder; it treats video as a sequence of images. The official reference pipeline extracts up to 50 evenly sampled frames per video, bypasses internal frame resizing, and passes them as a multi-image sequence inside a single chat message. That approach scored 27.14% on CinePile, positioning it between InternVL2 (2B) and Video-LLaVA (7B) on cinematic video understanding, a strong result given the model's size and that video was not the only thing it was trained for.

Setting Up the Environment

Hardware requirements:

Python packages:

# Python 3.10+ required
python --version
python -m venv smolvlm2-env
source smolvlm2-env/bin/activate  # macOS / Linux
smolvlm2-env\Scripts\activate     # Windows

# Install from the stable SmolVLM-2 branch -- required for SmolVLM2 support
pip install git+https://github.com/huggingface/transformers@v4.49.0-SmolVLM-2

# Core dependencies
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install \
  opencv-python \
  Pillow \
  numpy \
  num2words \
  accelerate

# Flash Attention 2 for CUDA -- significantly faster on NVIDIA GPUs
# Skip this on Apple Silicon and CPU -- it is CUDA-only
pip install flash-attn --no-build-isolation

# decord -- required for SmolVLM2's native video input path (used in Section 5)
pip install decord

Note: The num2words package is a non-obvious dependency. SmolVLM2's processor uses it to convert numeric digits to word representations (e.g. 3 → "three") for consistency with natural language training patterns, as explained in this walkthrough. Omitting it causes a silent import error when the processor loads.

Device check (run this before loading the model):

# device_check.py
# Run: python device_check.py
import torch

def detect_device():
    if torch.cuda.is_available():
        name = torch.cuda.get_device_name(0)
        vram = torch.cuda.get_device_properties(0).total_memory / 1e9
        print(f"CUDA: {name} ({vram:.1f} GB VRAM)")
        return "cuda", torch.bfloat16, "flash_attention_2"
    elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
        print("Apple Silicon MPS detected")
        return "mps", torch.float16, "eager"
    else:
        print("CPU fallback (slow -- consider Colab T4)")
        return "cpu", torch.float32, "eager"

if __name__ == "__main__":
    device, dtype, attn = detect_device()
    print(f"Device: {device} | dtype: {dtype} | attn: {attn}")

Run it with:

python device_check.py

Building the Foundation of the Pipeline

Before SmolVLM2 sees anything, you need frames. The frame extractor converts a video file into a list of PIL (Python Imaging Library) images with timestamps attached, one pair per extracted frame.

Two modes matter for different use cases:

• Uniform sampling distributes frames evenly across the full video duration, guaranteeing coverage of everything regardless of content. This is the right choice for meetings and lectures where you cannot afford to miss a section.
• Keyframe sampling extracts frames only where the visual content changes significantly, such as scene cuts, a new slide, or a new speaker, which reduces the frame count and focuses attention on distinct moments. This is better for surveillance and highlight extraction.

# frame_extractor.py
# Prerequisites: pip install opencv-python Pillow numpy
# Usage: from frame_extractor import FrameExtractor

import cv2
import numpy as np
from PIL import Image

class FrameExtractor:
    """
    Extracts video frames as PIL Images for SmolVLM2 inference.
    Each extracted frame is paired with its timestamp in seconds.
    
    SmolVLM2 uses ~81 visual tokens per image. At 50 frames that is roughly
    4,050 image tokens -- the practical upper limit before VRAM pressure
    affects generation quality on consumer GPUs.
    """
    
    MAX_FRAMES = 50
    
    def __init__(self, max_frames: int = MAX_FRAMES):
        """
        Args:
            max_frames: Hard cap on extracted frames. Default 50 matches the
                       SmolVLM2 reference pipeline's tested upper limit.
        """
        self.max_frames = max_frames
    
    def uniform_sample(self, video_path: str) -> list[tuple[float, Image.Image]]:
        """
        Extract evenly spaced frames across the full video duration.
        
        Best for: meeting recordings, lectures, tutorials, course content.
        
        Returns:
            List of (timestamp_seconds, PIL_Image) in chronological order.
        """
        cap = cv2.VideoCapture(video_path)
        if not cap.isOpened():
            raise IOError(f"Cannot open video: {video_path}")
        
        total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
        fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
        n_extract = min(self.max_frames, total_frames)
        
        # Build frame indices spread evenly from first to last frame
        indices = np.linspace(0, total_frames - 1, n_extract, dtype=int)
        
        results = []
        for idx in indices:
            cap.set(cv2.CAP_PROP_POS_FRAMES, int(idx))
            ret, frame = cap.read()
            if not ret:
                continue
            timestamp = round(idx / fps, 2)
            rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            results.append((timestamp, Image.fromarray(rgb)))
        
        cap.release()
        return results
    
    def keyframe_sample(
        self, video_path: str, diff_threshold: float = 30.0
    ) -> list[tuple[float, Image.Image]]:
        """
        Extract frames where visual content changes significantly.
        
        Best for: surveillance, event detection, highlight extraction.
        
        Uses mean absolute pixel difference between consecutive grayscale
        frames as the change signal. When the diff exceeds diff_threshold,
        a new keyframe is recorded.
        
        Args:
            diff_threshold: Mean pixel difference to treat as a scene change.
                           30.0 works for most commercial content.
                           Lower = more sensitive, higher = fewer frames.
        
        Returns:
            List of (timestamp_seconds, PIL_Image) in chronological order,
            capped at self.max_frames.
        """
        cap = cv2.VideoCapture(video_path)
        if not cap.isOpened():
            raise IOError(f"Cannot open video: {video_path}")
        
        fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
        results = []
        prev_gray = None
        idx = 0
        
        while len(results) < self.max_frames:
            ret, frame = cap.read()
            if not ret:
                break
            
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
            
            if prev_gray is None:
                # Always include the first frame
                rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                results.append((round(idx / fps, 2), Image.fromarray(rgb)))
                prev_gray = gray
            else:
                diff = np.mean(np.abs(gray.astype(float) - prev_gray.astype(float)))
                if diff > diff_threshold:
                    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                    results.append((round(idx / fps, 2), Image.fromarray(rgb)))
                    prev_gray = gray
            
            idx += 1
        
        cap.release()
        return results

Start every new video type with uniform_sample. If you find too many redundant frames (five nearly-identical slides in a row), switch to keyframe_sample and tune diff_threshold down from 30 to 20 until the extracted set feels representative without being redundant.

Loading SmolVLM2 and Running Single-Frame Inference

With frames in hand, here is the complete model loading and first-inference pattern. The important details: AutoModelForImageTextToText is the correct class (not the generic AutoModelForCausalLM), and on CUDA, you should enable Flash Attention 2, which provides meaningful latency improvements on multi-image inputs.

# smolvlm2_loader.py
# Prerequisites: transformers from v4.49.0-SmolVLM-2 branch, torch, flash-attn (CUDA only)
# Run: python smolvlm2_loader.py your_video.mp4

import sys
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

MODEL_ID = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"

def load_model():
    """
    Load SmolVLM2-2.2B and its processor.
    Automatically selects Flash Attention 2 on CUDA, eager mode elsewhere.
    First run downloads ~4.5 GB of weights to ~/.cache/huggingface/hub.
    """
    if torch.cuda.is_available():
        dtype = torch.bfloat16
        device = "cuda"
        attn = "flash_attention_2"
    elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
        dtype = torch.float16
        device = "mps"
        attn = "eager"
    else:
        dtype = torch.float32
        device = "cpu"
        attn = "eager"
    
    print(f"Loading {MODEL_ID} on {device}...")
    processor = AutoProcessor.from_pretrained(MODEL_ID)
    model = AutoModelForImageTextToText.from_pretrained(
        MODEL_ID,
        torch_dtype=dtype,
        _attn_implementation=attn,
    ).to(device)
    model.eval()
    print(f"Model ready on {device}")
    return model, processor

def describe_frame(
    model,
    processor,
    frame: Image.Image,
    prompt: str = "Describe what is happening in this frame in detail. Note any text, people, objects, or actions visible.",
    max_new_tokens: int = 256,
) -> str:
    """
    Run SmolVLM2 inference on a single PIL Image.
    
    The chat template expects image content before text content in the message --
    this mirrors the training data format and is important for reliable output.
    
    Args:
        frame: A PIL Image (from FrameExtractor)
        prompt: What to ask the model about this frame
        max_new_tokens: Maximum response length in tokens
    
    Returns:
        Model response as a plain string
    """
    messages = [
        {
            "role": "user",
            "content": [
                # Image placed before text -- matches SmolVLM2 training format
                {"type": "image"},
                {"type": "text", "text": prompt},
            ],
        }
    ]
    
    # apply_chat_template formats the message and injects visual token placeholders
    input_text = processor.apply_chat_template(
        messages, add_generation_prompt=True,
    )
    
    inputs = processor(
        images=[frame],
        text=input_text,
        return_tensors="pt",
    ).to(model.device)
    
    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,  # Greedy decoding for consistent structured output
        )
    
    # Decode only the newly generated tokens -- strip the input prompt
    new_tokens = output_ids[0][inputs["input_ids"].shape[-1]:]
    return processor.decode(new_tokens, skip_special_tokens=True).strip()

# ── Quick sanity check ────────────────────────────────────────────────────────
if __name__ == "__main__":
    from frame_extractor import FrameExtractor
    
    if len(sys.argv) < 2:
        print("Usage: python smolvlm2_loader.py <video_path>")
        sys.exit(1)
    
    video_path = sys.argv[1]
    model, processor = load_model()
    extractor = FrameExtractor(max_frames=5)
    
    frames = extractor.uniform_sample(video_path)
    print(f"Extracted {len(frames)} frames from {video_path}")
    
    for ts, img in frames[:3]:
        desc = describe_frame(model, processor, img)
        print(f"\n[{ts:.1f}s] {desc}")

Building the Synthesis Pipeline

Running inference on 50 frames individually gives you 50 descriptions, but what you actually want is a single coherent summary of the entire video. The synthesis pipeline handles this in two passes:

1. First pass: Run describe_frame on each extracted frame to produce per-frame descriptions.
2. Second pass: Concatenate all descriptions with their timestamps into a single prompt, and ask the model to synthesize a structured report.

# synthesis_pipeline.py
# Prerequisites: smolvlm2_loader.py, frame_extractor.py
# Usage: from synthesis_pipeline import synthesize_video

import re
from dataclasses import dataclass
from typing import Optional
from PIL import Image

@dataclass
class FrameDescription:
    """Structured output for a single frame analysis."""
    timestamp: float
    description: str

@dataclass
class VideoSummary:
    """Complete video summary output."""
    narrative: str
    action_items: list[str]
    key_moments: list[tuple[str, str]]  # (timestamp, description)
    raw_descriptions: list[FrameDescription]

def build_synthesis_prompt(
    descriptions: list[FrameDescription],
    duration: float,
) -> str:
    """Build the second-pass prompt that synthesizes frame descriptions into a report."""
    frames_text = "\n".join(
        f"[{int(d.timestamp // 60):02d}:{int(d.timestamp % 60):02d}] {d.description}"
        for d in descriptions
    )
    
    return f"""Below are time-stamped descriptions of frames from a {duration:.0f}-second meeting recording.

{frames_text}

Based on these descriptions, provide:

1. NARRATIVE SUMMARY: A 3-5 sentence summary of what the meeting covered, who participated (if visible), and what decisions or conclusions were reached.

2. ACTION ITEMS: A bullet list of concrete tasks or follow-ups mentioned or implied in the meeting. Start each with a dash (-).

3. KEY MOMENTS: A bullet list of the 3-5 most significant moments with their timestamps in [MM:SS] format.

Format your response with clear headings for each section."""

# ── Output parser ─────────────────────────────────────────────────────────────

def parse_action_items(text: str) -> list[str]:
    """Extract bullet-point action items from the synthesis output."""
    items = []
    for line in text.split("\n"):
        stripped = line.strip()
        if re.match(r"^[-*•]\s+", stripped) or re.match(r"^\d+\.\s+", stripped):
            clean = re.sub(r"^[-*•\d.]+\s*", "", stripped).strip()
            if clean and len(clean) > 5:
                items.append(clean)
    return items

def parse_key_moments(text: str) -> list[tuple[str, str]]:
    """Extract key moments with timestamps from the synthesis output."""
    moments = []
    pattern = r"\[(\d{2}:\d{2})\]\s*(.*)"
    for line in text.split("\n"):
        match = re.search(pattern, line)
        if match:
            moments.append((match.group(1), match.group(2).strip()))
    return moments