Do I Still Need a Monkey Patch for Gemini Live?

No. And deleting 187 lines of it was the single biggest benefit of moving to ADK 2.x - but it was not the only one, and it was not the last thing that needed fixing. What Does This Agent Do? The project is a biometric security scanner, built to exercise the parts of the Gemini Live API that a text chatbot never touches. A browser captures webcam and microphone, streams both to a FastAPI backend over a single WebSocket, and the backend forwards them to Gemini 3.1 Flash Live through the Agent Development Kit. The model watches the video feed, counts the fingers being held up, and calls a tool. Three tools are registered: - report_digit(count) - the detected finger count, which drives the UI - trigger_system_error() - fired on an offensive gesture, which terminates the session - trigger_heavy_metal_mode() - fired on the "Devil's Horns", a secret override The transport is deliberately plain. Binary WebSocket frames carry a 1-byte type prefix - 1 for audio, 2 for JPEG - with 16 kHz PCM going up and 24 kHz PCM coming back, played through an AudioWorklet so the main thread stays free. Everything runs locally with make run , or on Cloud Run behind make deploy . git clone https://github.com/xbill9/way-back-home cd way-back-home/level_3_new Two versions live side by side in that repo: level_3 , the original design, and level_3_new , the current one. Most of this article is the diff between them. What Level 3 Needed to Work The original build ran on google-adk 1.27.2, and it worked - but only because a file named patch_adk.py sat next to it, 187 lines long, applied at import time before anything else could run. The problem it solved was real. Gemini 3.1 deprecated media_chunks , the field a 1.x ADK used to send realtime media. A 1.x ADK talking to a 3.1 Live model would send the deprecated shape and get nothing useful back. The patch monkey-patched three separate call sites to translate: # level_3_gemini/backend/app/patch_adk.py if hasattr(rt_input, "media_chunks") and rt_input.media_chunks: logger.info("[PATCH] Unrolling 'media_chunks' from realtime_input.") for chunk in rt_input.media_chunks: ... await self.send_realtime_input(audio=chunk) ... await self.send_realtime_input(video=chunk) The three targets were live.AsyncSession.send_realtime_input , which unrolled media_chunks into the new typed keywords; GeminiLlmConnection.send_realtime , which routed each blob to audio= , video= or text= by mime type; and AudioCacheManager.cache_audio , which was guarded against a NoneType blob that would otherwise raise. It worked, and it was a liability. Monkey patching a framework means every upgrade is a gamble - the patch either becomes redundant, becomes wrong, or silently stops applying because the method it wraps was renamed. The closing recommendation in the original write-up was to delete it the moment the ADK supported the model natively. What ADK 2.x Handles Natively That moment arrived with google-adk 2.6.3, and patch_adk.py was deleted outright. The framework now does the routing itself, detecting the model generation and dispatching on it: # google/adk/models/gemini_llm_connection.py, send_realtime() if isinstance(input, types.Blob): if self._is_gemini_3_x_live or self._is_gemini_3_5_live_translate: if input.mime_type and input.mime_type.startswith('audio/'): await self._gemini_session.send_realtime_input(audio=input) elif input.mime_type and input.mime_type.startswith('image/'): await self._gemini_session.send_realtime_input(video=input) else: logger.warning( 'Blob not sent. Unknown or empty mime type for' ' send_realtime_input: %s', input.mime_type, ) else: await self._gemini_session.send_realtime_input(media=input) Note the third branch. Audio and image mime types are dispatched explicitly, and anything else is dropped with a warning rather than guessed at - so a blob sent with a missing or unexpected mime type goes nowhere, and the only evidence is a log line. Text is handled the same way. A single-part text Content is routed to send_realtime_input(text=...) for 3.x models rather than going out as client content, which matches the Live API's own guidance that send_client_content is only for seeding history. That is the whole first patch target and the whole second one, upstream, maintained, and tested by someone else. The third - the NoneType guard on cache_audio - was not carried over, because upstream still calls len(audio_blob.data) unguarded. No path in this application produces a blob with data=None , so it stays deleted rather than being reintroduced as a precaution. The One Thing That Broke The patch was hiding a bug in the calling code, and deleting it exposed the bug rather than causing it. LiveRequestQueue.send_realtime() accepts types.Blob and nothing else. The old patch had used model_construct internally, which skips Pydantic validation, so passing a bare string worked by accident. Without the patch it raises a ValidationError . The call site that matters is the keepalive. This project sends a text stimulus every ten seconds when the client goes quiet, and under 1.x that stimulus was a string handed straight to send_realtime() . Text has to go through send_content() instead: def send_text_stimulus(live_request_queue: LiveRequestQueue, text: str) -> None: live_request_queue.send_content( types.Content(role="user", parts=[types.Part(text=text)]) ) This is the removal most likely to take an agent off the air quietly. It does not fail at startup. It fails the first time the keepalive fires, ten seconds into a session that otherwise looks healthy. Anyone migrating a Live agent off 1.x should check that call site before touching anything else. What Else Was Updated The migration was the headline, but it was not the end of the work. Reading the Live API documentation with the source open beside it turned up several things that had been wrong the whole time, none of which any build, test or lint run had ever objected to. Video was running at twice the documented maximum. The capabilities guide is specific: Video frames are sent as individual images (e.g., JPEG or PNG) at a specific frame rate (max 1 frame per second). The project ran at 2 FPS and permitted up to 5 through an environment variable. Nothing rejects the surplus frames, which is why it went unnoticed - but they are billed, and they consume the session budget twice as fast. VIDEO_FPS now defaults to 1.0 and is hard-clamped there, so VIDEO_FPS=3 yields 1.0 rather than being honoured. A documented limit that is not enforced is how the 2 FPS crept in to begin with. Audio-plus-video sessions cap at two minutes. From the session management guide: audio-only sessions are limited to 15 minutes, and audio-video sessions are limited to 2 minutes Context window compression removes the cap entirely. RunConfig.context_window_compression defaults to None , so it has to be asked for: context_window_compression=types.ContextWindowCompressionConfig( sliding_window=types.SlidingWindow(), ), This application streams both continuously, so it had been on the two-minute clock since the first version. Short test sessions never reached it. A demo where someone works through five gestures does. Interruptions were documented and unhandled. When a user talks over the model, the model stops generating - but the audio it already sent is sitting in the client's ring buffer and keeps playing. The Live API guidance is to stop playback and clear the queue on interruption. ADK surfaces interrupted on the event, and because the backend forwards whole events as JSON, the flag was already arriving in the browser with nothing reading it. The clearing machinery already existed too. Three lines connected them. The Log Line That Never Ran Both input and output audio transcription were enabled in RunConfig from the very first version of this project. Neither ever produced a line of output. input_transcription = getattr(event, "input_audio_transcription", None) if input_transcription and input_transcription.final_transcript: logger.info(f"USER TRANSCRIPT: {input_transcription.final_transcript}") Two mistakes are stacked here. input_audio_transcription is the RunConfig field that enables transcription - it is not the field on the event that transcription produces. And final_transcript is not a member of types.Transcription at all; the fields are text , finished , language_code , speaker_label and words . Either mistake alone raises AttributeError and gets fixed in minutes. Together, behind the default on getattr , they produce silence. The condition evaluates to None and ... , which is falsy, forever. The correct field names: input_transcription = getattr(event, "input_transcription", None) if input_transcription and input_transcription.finished: logger.info(f"USER TRANSCRIPT: {input_transcription.text}") Gating on finished is deliberate. ADK emits partial transcription events with finished=False and one accumulated event with finished=True , so this produces one clean line per turn instead of one per fragment. The run_live() docstring is the reference: partial and non-partial events are both yielded to the caller, but only non-partial ones are saved to the session. The fix produces this on connect, which is the exact opening line the agent instruction specifies: INFO - GEMINI TRANSCRIPT: Scanner Online. Video That Stops When You Look Away The original design captured frames on a timer: intervalRef.current = setInterval(() => { /* capture, send */ }, 500); The rewrite replaced that with requestAnimationFrame and a manual elapsed-time check. On paper it is the better primitive - frame-aligned, idle when the compositor has nothing to do, and paired with toBlob instead of toDataURL it keeps JPEG encoding off the main thread. In a backgrounded tab, requestAnimationFrame is throttled to zero. The microphone is not. It runs in an AudioWorklet on the audio thread, which browsers keep alive so capture and playback survive a tab switch. The result is an asymmetric, silent failure: switch tabs and vi