This is how I added an in-browser auto captions feature to my YouTube Shorts converter web application using Whisper AI and ffmpeg.wasm

Overview

A few weeks ago I launched Convert to Shorts - a free browser-based tool that converts horizontal videos to YouTube Shorts format (9:16) without uploading anything to a server. The most requested feature after launch was auto captions. Captions significantly boost Shorts engagement since most people watch without sound, and manually typing captions is tedious. The challenge: how do you add free auto captions to a privacy-first tool that never uploads your video to a server? The answer: run Whisper AI in the browser.

The Stack

• Transformers.js (@xenova/transformers) - Hugging Face's JavaScript port of the Transformers library, runs ONNX models in the browser via WebAssembly
• Whisper tiny - OpenAI's speech recognition model, 75MB, surprisingly accurate for clear speech
• Web Audio API - for extracting and resampling audio from the video file
• ffmpeg.wasm - for burning captions into the video
• ASS subtitles - the subtitle format libass (inside ffmpeg.wasm) understands

Step 1: Audio Extraction

Whisper expects mono 16kHz audio as a Float32Array. The Web Audio API handles this cleanly:

async function extractAudio(file: File, trimStart: number, trimEnd: number): Promise<Float32Array> {
  const arrayBuffer = await file.arrayBuffer();
  const audioContext = new AudioContext({ sampleRate: 16000 });
  const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
  const sampleRate = audioContext.sampleRate;
  const startSample = Math.floor(trimStart * sampleRate);
  const endSample = Math.floor(trimEnd * sampleRate);
  // Mix down to mono, slice to trim range
  const channelData = audioBuffer.getChannelData(0);
  const trimmed = channelData.slice(startSample, endSample);
  await audioContext.close();
  return trimmed;
}

Creating the AudioContext at 16kHz means the browser automatically resamples from whatever the source rate is (usually 44.1kHz or 48kHz). No manual resampling needed.

Step 2: Running Whisper Transformers.js

const { pipeline, env } = await import("@xenova/transformers");
env.useBrowserCache = true;

const transcriber = await pipeline(
  "automatic-speech-recognition",
  "Xenova/whisper-tiny",
  {
    progress_callback: (p) => {
      if (p.status === "downloading") {
        setModelProgress(Math.round((p.loaded / p.total) * 100));
      }
    },
  }
);

const result = await transcriber(
  audioFloat32Array,
  {
    return_timestamps: true,
    chunk_length_s: 30,
    stride_length_s: 5,
  }
);

return_timestamps: true gives you back an array of segments with text and timestamp: [start, end] - exactly what you need for timed captions. env.useBrowserCache = true means the model downloads once and is stored in IndexedDB. Every subsequent use loads from cache - no 75MB download each time.

Step 3: Burning Captions with ffmpeg.wasm

My first instinct was to use ffmpeg's drawtext filter with an enable='between(t,start,end)' expression for each caption segment - one filter per caption, chained with commas. This failed in multiple ways:

• The comma inside between(t,start,end) was interpreted as a filter separator
• Escaping with \\ , didn't work in ffmpeg.wasm's argument parsing
• Using gte(t,start)*lte(t,end) instead of between also failed at the filter chain level
• Chaining 20+ drawtext filters caused ffmpeg.wasm to abort

The reliable solution was ASS subtitles. ASS (Advanced SubStation Alpha) is a subtitle format that ffmpeg's built-in ass filter handles natively via libass:

function buildAssSubtitles(segments, color, size) {
  function toAssTime(seconds) {
    const h = Math.floor(seconds / 3600);
    const m = Math.floor((seconds % 3600) / 60);
    const s = Math.floor(seconds % 60);
    const cs = Math.round((seconds % 1) * 100);
    return `${h}:${String(m).padStart(2,"0")}:${String(s).padStart(2,"0")}.${String(cs).padStart(2,"0")}`;
  }

  const header = [
    `[Script Info] ScriptType: v4.00+ PlayResX: 1080 PlayResY: 1920 WrapStyle: 1 ScaledBorderAndShadow: yes [V4+ Styles] Format: Name, Fontname, Fontsize, PrimaryColour, ... Style: Default,Roboto Bold,75,&H00FFFFFF,... [Events] Format: Layer, Start, End, Style, Text`
  ];

  const events = segments.map(seg =>
    `Dialogue: 0,${toAssTime(seg.start)},${toAssTime(seg.end)},Default,${seg.text}`
  ).join("\n");

  return header + events;
}

Step 4: The Font Problem

libass needs a font to render subtitles. In a normal environment it uses system fonts. In ffmpeg.wasm's WebAssembly sandbox there are no system fonts. Three approaches were tried:

Approach 1: Embed font as Base64 in the ASS file

The ASS format supports a [Fonts] section with Base64-encoded font data split into 80-character lines. This failed with a libass assertion error in ass.c about Base64 padding - the ffmpeg.wasm build of libass appears to have a bug in its font decoder.

Approach 2: Write font to ffmpeg.wasm virtual filesystem and use fontsdir

Write the font file to /fonts/Roboto-Bold.ttf in the virtual filesystem before running ffmpeg. This worked:

await ffmpeg.createDir("/fonts");
const fontResponse = await fetch("/Roboto-Bold.ttf");
const fontBuffer = await fontResponse.arrayBuffer();
await ffmpeg.writeFile("/fonts/Roboto-Bold.ttf", new Uint8Array(fontBuffer));

The key insight: ffmpeg.wasm has a full virtual filesystem (Emscripten's FS). You can create directories and write files to it just like a real filesystem, and ffmpeg commands can reference those paths. As a result the full caption pipeline added roughly 10-30 seconds to the export time for a 30-60 second clip.

Lessons Learned

• Transformers.js is genuinely production-ready. The API is clean, browser caching just works, and the ONNX runtime handles the WebAssembly execution reliably.
• ASS subtitles are more robust than drawtext filter chains.
• If you're burning timed text into video with ffmpeg.wasm, reach for the ass filter before trying to chain multiple drawtext filters.
• ffmpeg.wasm's virtual filesystem is powerful. You're not limited to just reading and writing video files - you can create directory structures, write fonts, write subtitle files, and reference them all from ffmpeg commands exactly as you would on a real filesystem.
• libass has quirks in the WASM build. The embedded font Base64 decoding in the WASM build of libass appears broken. Use fontsdir instead.

Try it at converttoshorts.com - free, no account, no upload. Auto captions are in the export panel.