Don’t Send the Whole Camera: Build One-Shot Visual Context for a Tencent RTC Voice Companion

A multimodal voice companion creates an awkward product tension: users want to ask “What am I looking at?” without granting an AI system indefinite access to their camera. The easiest implementation-forwarding frames continuously-also creates hidden costs. It increases data transfer and model work, makes visual context harder to reproduce, and leaves users unsure when the companion is actually observing them. It can also produce a subtler correctness bug: the model answers from an old frame while speaking as if it can see the present. A better default for many companion experiences is one-shot visual context: - The user deliberately shares one frame. - The application records when and why it was captured. - At most one voice turn can consume it. - Expired or withdrawn frames never reach the model. - If visual input is unavailable, the companion says so instead of guessing. This tutorial builds that control layer in TypeScript. Tencent RTC supplies the real-time conversational setting, while Gemini sits behind an application-owned multimodal model port. We will not treat the model as the camera controller, consent authority, speech recognizer, or media transport. First decide whether a snapshot is enough Multimodal capability does not automatically justify continuous vision. Choose the smallest visual scope that supports the task. | User task | Visual policy | Trade-off | |---|---|---| | Identify an object or read a label | One approved frame | Low exposure, but the user may need to recapture | | Compare two arrangements | Two explicitly labelled frames | More application state and UI work | | Explain ongoing movement | Time-bounded video may be necessary | Higher privacy, bandwidth, and moderation cost | | General voice companionship | Camera off by default | The companion cannot answer visual questions until invited | A still frame is the wrong abstraction for motion. If someone asks whether their exercise form remains correct over ten seconds, do not send one image and let the model imply that it observed the full movement. The demonstrated capability is narrower: a multimodal model can receive a bounded text-and-image request. The hype-shaped interpretation-that it continuously understands the user’s environment-is a product decision your application should not silently make. Keep the pipeline boundaries visible A production conversational pipeline should remain separable: Microphone -> RTC/media transport -> speech recognition -> application turn coordinator -> approved visual snapshot -> Gemini/model adapter -> output moderation -> speech synthesis -> RTC/media transport -> user Tencent Conversational AI is documented as a real-time voice interaction scenario that can work with multiple LLM providers. Its overview is the appropriate starting point for the voice architecture: Tencent RTC also documents LLM configuration, including OpenAI-compatible model connections, agent platforms such as Dify or Coze, and request identifiers used for routing and observability: Do not infer from voice connectivity that a selected model route accepts images. Validate multimodal support for the exact provider and model configuration you operate. A text-only route should fail as text-only, not quietly discard the image and produce a confident answer. Define the interaction states The visual permission and the voice turn are related, but they are not the same state. Our snapshot can be: off -> capturing -> ready -> consumed off -> capturing -> unavailable capturing/ready/requesting -> off, when the user withdraws it A monotonically increasing visualEpoch invalidates asynchronous work. Every capture and withdrawal advances the epoch. A callback may mutate state only if it still belongs to the current epoch. The important invariants are: - A snapshot is used by no more than one turn. - A snapshot older than the freshness limit is rejected. - Withdrawal invalidates pending capture and model work. - A new snapshot invalidates an image-based request using the old one. - A visually dependent question without a valid image produces a clarification, not a guess. - Model output passes an application-controlled output gate before synthesis. Create the project mkdir one-shot-visual-companion cd one-shot-visual-companion npm init -y npm install --save-dev typescript tsx @types/node npx tsc --init mkdir src Add these scripts to package.json : { "scripts": { "check": "tsc --noEmit", "test": "tsx --test src/*.test.ts" } } Use a strict TypeScript configuration: { "compilerOptions": { "target": "ES2022", "module": "NodeNext", "moduleResolution": "NodeNext", "strict": true, "noUncheckedIndexedAccess": true } } Implement the one-shot coordinator Create src/visual-turn.ts : import { randomUUID } from 'node:crypto'; export type Snapshot = { id: string; capturedAt: number; mimeType: 'image/jpeg' | 'image/png'; bytes: Uint8Array; }; type VisualState = | { kind: 'off' } | { kind: 'capturing'; epoch: number } | { kind: 'ready'; epoch: number; snapshot: Snapshot } | { kind: 'unavailable'; epoch: number; reason: string }; export interface CameraPort { captureOneFrame(): Promise ; } export type ModelRequest = { requestId: string; text: string; image?: { mimeType: Snapshot['mimeType']; bytes: Uint8Array; }; signal: AbortSignal; }; export interface MultimodalModelPort { generate(request: ModelRequest): Promise ; } export interface OutputGate { approve(text: string): Promise ; } export interface CompanionUi { status(message: string): void; speakApprovedText(text: string): void; } type ActiveRequest = { turnId: string; visualEpoch: number; usedImage: boolean; abort: AbortController; }; export class VisualTurnCoordinator { private visualEpoch = 0; private visual: VisualState = { kind: 'off' }; private active?: ActiveRequest; constructor( private readonly camera: CameraPort, private readonly model: MultimodalModelPort, private readonly outputGate: OutputGate, private readonly ui: CompanionUi, private readonly now: () => number = Date.now, private readonly maxSnapshotAgeMs = 15_000 ) {} async shareOneFrame(): Promise { this.abortImageRequest('A newer visual context was requested.'); const epoch = ++this.visualEpoch; this.visual = { kind: 'capturing', epoch }; this.ui.status('Capturing one frame…'); try { const snapshot = await this.camera.captureOneFrame(); // The user may have withdrawn permission while capture was pending. if (epoch !== this.visualEpoch) return; this.visual = { kind: 'ready', epoch, snapshot }; this.ui.status('One frame is ready for your next question.'); } catch (error) { if (epoch !== this.visualEpoch) return; const reason = error instanceof Error ? error.message : 'Capture failed'; this.visual = { kind: 'unavailable', epoch, reason }; this.ui.status('I could not capture the frame. Voice mode is still available.'); } } stopSharing(): void { ++this.visualEpoch; this.visual = { kind: 'off' }; this.abortImageRequest('Visual sharing was withdrawn.'); this.ui.status('Visual context is off.'); } async onFinalTranscript(text: string): Promise { const trimmed = text.trim(); if (!trimmed) return; const selected = this.takeFreshSnapshot(); if (refersToVisibleContext(trimmed) && !selected) { this.ui.status('I do not have a current frame. Share one frame, or describe it aloud.'); return; } const turnId = randomUUID(); const abort = new AbortController(); const visualEpoch = this.visualEpoch; this.active = { turnId, visualEpoch, usedImage: selected !== undefined, abort }; this.ui.status(selected ? 'Thinking about the approved frame…' : 'Thinking…'); try { const answer = await this.model.generate({ requestId: turnId, text: trimmed, image: selected ? { mimeType: selected.mimeType, bytes: selected.bytes } : undefined, signal: abort.signal }); if (!this.isCurrent(turnId, visualEpoch, selected !== undefined)) return; if (!(await this.outputGate.approve(answer))) { if (this.active?.turnId === turnId) this.active = undefined; this.ui.status('That response could not be played. Please rephrase the question.'); return; } if (!this.isCurrent(turnId, visualEpoch, selected !== undefined)) return; this.active = undefined; this.ui.speakApprovedText(answer); } catch (error) { if (this.active?.turnId !== turnId) return; this.active = undefined; if (abort.signal.aborted) { this.ui.status('That visual turn was cancelled.'); } else { this.ui.status('The model could not answer. You can retry without sharing another frame.'); } } } debugVisualKind(): VisualState['kind'] { return this.visual.kind; } private takeFreshSnapshot(): Snapshot | undefined { if (this.visual.kind !== 'ready') return undefined; const { snapshot } = this.visual; this.visual = { kind: 'off' }; // Consume it before starting asynchronous work. if (this.now() - snapshot.capturedAt > this.maxSnapshotAgeMs) { this.ui.status('That frame expired. Share a new one if the question is visual.'); return undefined; } return snapshot; } private abortImageRequest(message: string): void { if (!this.active?.usedImage) return; this.active.abort.abort(message); this.active = undefined; } private isCurrent( turnId: string, visualEpoch: number, usedImage: boolean ): boolean { if (this.active?.turnId !== turnId) return false; if (usedImage && visualEpoch !== this.visualEpoch) return false; return true; } } function refersToVisibleContext(text: string): boolean { return /\b(this|that|these|those|here|camera|in front of me)\b/i.test(text); } The regular expression is intentionally not an AI safety classifier. It only catches obvious phrases that would otherwise invite a visual guess. False positives should result in a clarification, not an unsafe action. Notice that image bytes leave the coordinator only inside MultimodalModelPort.generate . This gives the application one auditable transfer boundary. It does not prove that an upstream provider deletes the image; retention, regional processing, and provider logging still need to be communicated and configured separately. Put Gemini behind an internal contract Avoid spreading