Holding Many Modalities in One Session: A Parts-Based API Structure

Parts, Not Endpoints

The mental shift that made this tractable: stop thinking in endpoints, start thinking in typed parts. A request is an ordered list where each element declares its own type:

type Part =
  | { kind: 'text'; text: string }
  | { kind: 'image'; mime: string; data: Uint8Array | { uri: string } }
  | { kind: 'audio'; mime: string; data: Uint8Array }
  | { kind: 'video'; mime: string; uri: string }

Once input is a part list, interleaving stops being a special case. "Here are two images, a question about them, and an audio clip for tone reference" is just a four-element array - not three API calls you have to correlate afterwards.

Inline Versus Reference, and Why It Bites

Small assets go inline as base64. Large ones must be uploaded first and passed by URI. The threshold is provider-specific and the failure mode is bad: exceed it and you get an opaque 400, not a helpful "too large, upload it instead." Worth building a size check into your part constructor so it routes automatically rather than discovering the limit in production.

Also: base64 inflates payloads by roughly a third. A "small enough to inline" image is meaningfully smaller than you assume.

Streaming Breaks the Tidy Model

Text streams token by token. Images arrive whole. Audio may stream as chunks. So your response handler cannot assume a uniform shape - it needs to be a state machine over part-typed events, accumulating text deltas while treating an image part as atomic. The bug I hit repeatedly: treating every stream event as appendable text, which quietly corrupts binary parts.

Context Accumulation Is the Real Cost

Multi-turn multimodal sessions grow fast. Every image stays in context and keeps costing tokens on every subsequent turn. Two things that helped:

• Summarise old turns - after N turns, replace early image parts with a text description of what they were
• Explicit pinning - let the user mark which assets stay in context, and drop the rest

Without either, a ten-turn session with images becomes surprisingly expensive, and the cost is invisible until the bill arrives.

I put this architecture into practice at GeminiOmni - image editing, video generation, and live chat sharing one session model.

Caveat

Part-list APIs are converging across providers but are not identical. This structure ports with modest adapter work - it is not free.