[Dev Log][Python] Create short videos from photos and clips with Gemini 3.7 Flash: ReelCraft

Preface

It all started with a misunderstanding. I noticed a new page in the Gemini API documentation called Omni, introducing a model named Gemini Omni Flash, described as "natively multimodal, processing text, images, audio, and video simultaneously."

My first thought was straightforward: if I throw a whole folder of videos and photos from my phone into it, let it understand what each asset is about, and then tell it in one sentence to edit them into a short video-isn't that a video editing app?

After reading the documentation, I realized I had misunderstood, and the misunderstanding happened to be at the most critical point. However, after bypassing that limitation, the rest was actually feasible.

The result is ReelCraft: a Python CLI where you feed in a bunch of videos and photos, Gemini 3.7 Flash understands the assets one by one and provides editing suggestions. Once I confirm the edit list, ffmpeg cuts it into a 9:16 vertical short video, background music is generated using Lyria 3, and subtitles are automatically burned in.

Along the way, there were three issues where both ffmpeg and Gemini reported success, but the output was wrong-the kind of errors you only discover by actually playing the video.

TL;DR

This article will cover:

• Omni Flash is not what I thought it was
• Bypassing limitations: Per-file understanding, then text aggregation
• Using edl.yaml as a human confirmation point
• The difference after switching to Gemini 3.7 Flash
• Background music: Lyria 3 uses a different API
• ffmpeg will silently fail your edits
• Subtitles: Two issues only visible after burning them in
• Other pitfalls
• Conclusion
• Reference links

Omni Flash is not what I thought it was

Gemini Omni Flash (gemini-omni-flash-preview) is a video generation and editing model that uses the Interactions API. It allows you to use natural language to apply effects to a single video, such as "when the person touches the mirror, make the mirror ripple beautifully like liquid."

It is not a tool for "understanding a bunch of videos."

The limitation section states clearly:

Referencing or reasoning across multiple videos is not supported. Attempting multi-video prompting may result in degraded model performance or unexpected outputs.

Additionally:

Video references up to 3 seconds in duration are accepted by the API schema but are not correctly processed by the model at this time.

So the path of "throwing a bunch of videos in and letting it understand and edit them" was blocked for Omni Flash.

The models that can actually perform multi-video understanding are the standard Gemini models: starting from version 2.5, a single request can include up to 10 videos. With a 1M context window, it can handle about an hour of footage at default resolution, tokenize it second-by-second, and output scene descriptions with timestamps.

The time spent on this misunderstanding wasn't wasted. The verification process helped clarify "which task should be handled by which model," and the architecture followed naturally.

Bypassing limitations: Per-file understanding, then text aggregation

The entire pipeline is split into five stages, with states stored in files:

[Asset Folder]
│
poc ingest: Scan videos/photos → catalog.json
│
▼
poc analyze: Call Gemini for each file individually → analysis/*.json
│
▼
poc plan: Aggregate all analysis results, call once for editing suggestions
│
▼ → summary.md (for humans) + edl.yaml (for machine execution)
⏸ Human inspection and editing of edl.yaml
│
▼
poc render: ffmpeg editing, 9:16 cropping, xfade transitions
▼
output/final.mp4

The key design decision is in the second and third steps: call Gemini once for each video to get precise internal timestamps and descriptions; then feed these text results (not the raw videos) into a second call for cross-asset aggregation, sequencing, and editing suggestions.

This approach has two benefits:

1. It completely avoids the "multi-video reasoning not supported" issue because the second call only sees text, not ten videos.
2. It isn't limited by the 10 videos/request cap; no matter how many assets there are, it just means more independent calls in the analyze phase. Those calls can be retried or fail individually without affecting each other.

Testing also proved that timestamps are more reliable when processed separately. When asking about ten videos in a single prompt ("which seconds are the highlights?"), the model easily confuses the timelines of different videos.

Failure handling in the analyze phase is recorded separately: if a file fails after three retries, it's logged in analysis/_errors.json, while other files continue. This later revealed a loophole during review, which I'll discuss later.

Using edl.yaml as a human confirmation point

I decided from the start not to make it "one-click fully automatic." Between inputting assets and outputting the final product, there must be a place where I can manually intervene, because LLM-provided edit points will inevitably have some irrationalities, and re-running the entire pipeline incurs API costs again.

That interface is a YAML file:

target_duration_sec: 23
aspect_ratio: '9:16'

clips:
  - source: /abs/path/808327978.mp4
    note: "Opening shot: Showing the COSCUP x UbuCon Asia main visual backdrop."
    in: '00:00.000'
    out: '00:02.500'

  - source: /abs/path/S__1908753.jpg
    note: "Fun venue easter egg: Creative semiconductor chip snacks distributed on-site."
    duration_sec: 4.0

transitions:
  crossfade 0.3s

mood_tags: [Professional, Joyful, Community Cohesion]

Videos use in / out to mark the range, photos use duration_sec for duration, and note is the reason for selection written by Gemini (this field was later used for subtitles, see below).

To change an edit point, just change the numbers; to change the order, move the clip; after saving, run poc render.

The outputs of each stage remain in the project directory, so any step can be re-run individually. analyze also skips files that already have analysis results, so re-running doesn't incur double charges-this is very helpful when iterating on prompts.

poc plan --theme was added later: you can provide a sentence as the editing theme, e.g., --theme "Participating in the COSCUP open source community". This affects the narrative angle of the summary, the priority of clip selection, and the wording of each clip's note. Since it only affects the plan stage, changing the theme doesn't require re-analyzing assets, making it very cheap to try different narratives on the same set of materials.

The difference after switching to Gemini 3.7 Flash

The understanding and aggregation stages initially used gemini-2.5-flash, then switched to gemini-3.7-flash. This is the GA stable version, not a preview:

For this project, the most important features are structured outputs and video input, as the analyze stage involves feeding in a video and requesting a JSON with a fixed schema.

After switching, I didn't just change the string and call it a day; I verified it with actual API calls, running analyze_file on real assets. For the same lecture video, the difference in descriptions between the two models was quite noticeable.

gemini-2.5-flash version:

At the start of the video, a woman on stage uses a microphone to introduce herself to the audience. The large screen behind her shows her name "Zona Wang" and her job description.

gemini-3.7-flash version:

In the video, a female speaker (Zona Wang, LINE Technology Evangelist) is giving a self-introduction and presentation on a stage in a lecture hall, followed by a camera pan across the audience listening intently.

The difference lies in "job description" vs. "LINE Technology Evangelist." The latter actually read the small text on the slide, while the former only knew there was some job information there.

The gap in the aggregation stage was even larger. For the same set of COSCUP assets and the same --theme, 2.5's summary was:

"This short video aims to showcase the vitality and diversity of the COSCUP open source community. From professional knowledge sharing and deep technical exchange to warm interaction and inclusion among community members"

-the whole thing stayed at an abstract level.

3.7 recognized the full event name "COSCUP x UbuCon Asia," booth names like "FOSS for All" and "Kubernetes," and even described a photo as "Creative semiconductor chip snacks distributed on-site." These details weren't in my prompt; they all came from the text and objects in the photos.

For an application where "asset understanding quality directly determines editing quality," the benefit of switching models was greater than I expected. The editing suggestions improved because it actually understood more, not because the prompt was written better.

By the way, 3.7's note style also changed to a "Short Label: Detailed Description" format. This change later broke all my subtitles, as discussed below.

Background music: Lyria 3 uses a different API

Background music is generated using Lyria 3. There are two models:

• lyria-3-clip-preview for 30-second clips
• lyria-3-pro-preview for full songs

My output is about 20 seconds, so the clip version is perfect. It doesn't require a separate Vertex AI application or allowlisting; the same Gemini API key works.

However, the calling method is completely different from generate_content, using client.interactions.create():

interaction = client.interactions.create(
    model="lyria-3-clip-preview",
    input="An instrumental background music track for a short social-media video, "
          "about 20 seconds long. Mood: Professional, Joyful, Community Cohesion, Happy. "
          "No vocals, no lyrics, loopable.",
)

audio_bytes = base64.b64decode(interaction.output_audio.data)

Several things were different from what I imagined:

• It has no structured parameters. Length, BPM, genre, and mood must all be written in the natural language prompt, rather than passing a field like bpm=120. So the generate_score(mood_tags, duration_sec) function's job is actually to concatenate mood tags and seconds into an English sentence.
  • Mood tags are aggregated from asset analysis results during the plan stage, and poc render --mood "Happy, Joyful, Celebration" can further overlay desired directions.
• It is single-turn generation and cannot be iteratively modified. Unlike Omni Flash's video editing, once the music is generated, it's set; if you're not satisfied, you have to submit a new prompt.
• All generated audio includes a SynthID watermark.
• When the music is shorter than the video, you have to handle it yourself. The clip version is max 30 seconds, but the video might be longer. So during mixing, I use -stream_loop -1 to loop the audio infinitely and -shortest to trim it to the video length:

cmd.extend(["-stream_loop", "-1", "-i", str(audio_path)])
# ... filter_complex, map video ...
cmd.extend(["-map", f"{audio_index}:a", "-c:a", "aac", "-b:a", "128k", "-shortest"])

Music generation failure (quota, network, safety filters) won't crash the entire render; it prints a warning and falls back to silent output. This principle was later added to the project's CLAUDE.md: any value-added feature calling an external generative API must degrade gracefully and not let the main process die because of a secondary feature.

ffmpeg will silently fail your edits

The render stage uses ffmpeg's xfade filter to connect clips. Each xfade requires an offset parameter, which is "at which second in the output timeline to start this transition." The logic for accumulation is: the sum of all previous clip lengths minus the seconds overlapped by each transition.

After writing the first version, unit tests were all green, and real assets produced normal videos. Then review identified two scenarios where ffmpeg returns exit code 0, but the output file is wrong.

Scenario one: The transition is longer than the clip, causing the clip to be silently swallowed

For two 1-second clips with transitions: "crossfade 2s", the calculated offset is -1.000. ffmpeg accepts this negative number, doesn't report an error, and finishes normally. The output is a 1-second video containing only the first clip; the second one disappears entirely.

Since EDL.transitions is a free-text field, it's entirely possible for me to type 3s instead of 0.3s when manually editing the YAML, and it won't tell me in any way.

Scenario two: out exceeds the actual asset length, causing everything following it to be truncated

For a 10-second video, if the EDL says in: 8.0 / out: 15.0, only 2 seconds can actually be taken. If a 1.5-second photo follows, the offset is calculated as 6.700, which falls after the end of the first stream. The result is a 2-second output where the photo is completely missing, and the exit code is still 0.

This scenario is even more important to prevent because the EDL is generated by an LLM, and hallucinating an out-of-bounds end time is quite natural.

I added explicit checks for both:

• If a negative offset is calculated, a ValueError is thrown specifying which clip and transition length.
• Before rendering, ffprobe is used to read the actual length of each video asset, and if out exceeds it, an error is reported clearly stating the requested vs. actual duration.

I care so much because a "successful" but incorrect output is much worse than a crash. If it crashes, I know to fix it immediately. With exit code 0 and a seemingly normal mp4, I might not notice until I watch the whole video and think "wait, a segment is missing," and then have no idea where to start investigating.

Subtitles: Two issues only visible after burning them in

The source for subtitles is the note for each clip in the EDL-the editing reason writ