Start and end a voice agent call in Python without clipping a word

Every phone call has two fragile boundaries: the pickup and the hang-up. In real-time voice agents, these edges are where conversations lose words. A caller starts speaking the millisecond they hear the line connect, often well before your client application has finished handshaking with the upstream speech API. Minutes later, as the conversation wraps up, the agent begins saying goodbye, only for the client to assume the exchange is over and abruptly sever the socket mid-sentence. To see how words get lost at the edges, look at this scripted call from the companion repository, pmckeetx/barge-in-voice-agent, connecting to PolyAI's Dialog-RSN-1 API. Here, an early client implementation handles a routine hotel booking inquiry, stumbling at both ends: (caller says pickup.wav) (session ready: sent 0.00 s of held audio, now live) Caller: Yes, I'm calling about my booking for Friday. The surname's Okafor. (look_up_booking returned for Okafor) Agent: Let me pull up your booking, the system can take a moment to load. Agent: I have your booking here, Ms. Okafor. You're in room 214 for two nights, checking in on Friday. Is there anything else you need to know? (caller says thats-all.wav) Caller: Great, that's everything I needed. Thanks very much. (end_call: Caller confirmed they had everything they needed and said goodbye.) (hangs up) first line: lost 1 of 12 words at the start (session ready 0.26 s after pickup); goodbye: played 1.0 s of "You're welcome, Ms. Okafor. Have a wonderful stay with us. Goodbye.", then hung up ... Notice the damage on both sides of the exchange. The agent heard "Yes, I'm calling about..." because the caller's initial "Hi" landed during the quarter-second window before the session was ready, and the client discarded it. At the end of the call, the agent composed a polite farewell, but the client slammed the line down after a static one-second sleep while the server was still streaming audio chunks over the wire. There isn't even an Agent: transcript line for the goodbye because the call ended before the reply did. Neither failure is the model's fault. The speech recognizer is working fine, and the language model knows what to say. The breakdown happens entirely in client-side lifecycle management: the client assumes the call hasn't started until the network is ready, and assumes the call is finished before playback concludes. To fix this, we can equip our front-desk agent (the Harborview Inn from the repository) with two changes: a circular audio buffer that starts capturing frames the moment the sound card activates, flushing them the instant the session is configured; and an event-driven hang-up routine that tracks the server's response lifecycle and drains the playback buffer before tearing down the connection. When benchmarked against the live API on 10 November 2026 across 10 runs each, the naive client clipped the caller's first word on every single call and severed every goodbye. The buffered, event-driven client captured every word at both ends. Code: pmckeetx/barge-in-voice-agent (call.py and pickup.py ) What you'll build You will build call.py , a voice agent client that listens from the exact instant the line opens and refuses to hang up until its farewell has finished playing through the speakers. Under the hood, it: - Holds up to 5 seconds of caller audio in memory while waiting for the session handshake. - Exposes an end_call tool configured with a spoken goodbye argument. - Waits up to 8 seconds for that goodbye to fully stream and play. - Cancels the hang-up and keeps the line open if the caller talks over the goodbye. To verify these behaviors reliably, you'll also use pickup.py , a test harness that feeds scripted caller audio into the pipeline from the very first frame without requiring a live microphone. Here are the old client and the new one on the same scripted call: How it works To understand why calls clip, we have to look closely at the mechanics of the session lifecycle. A call begins with an initialization dance across the network. The client opens a WebSocket, the server answers with session.created once accepted, the client replies with session.update specifying tools and audio configurations, and the server confirms with session.updated . Until that final confirmation arrives, the session runs on its defaults (which on 10 November 2026 meant raw 24 kHz PCM input, per the quickstart). In an isolated connection probe, that handshake is fast: session.updated arrived a median 234 ms after the connection began (measured over 20 runs on api.studio from the US Midwest). In the 20 scripted calls further down, sessions were ready 0.23 to 2.83 seconds after pickup (median 0.66 s), for reasons I didn't pin down. A caller saying "Hi, yes..." gets those first words out inside that window, and routing calls over mobile networks or SIP trunks only widens the gap. The end of the call presents the inverse problem. When the model decides the interaction is complete, it triggers end_call . Rather than sending plain text, the server speaks the tool's goodbye argument directly. The tool invocation and the outgoing audio stream arrive simultaneously: in 10 test calls, the first audio chunk arrived a median 46 ms before the client received the end_call tool event. The server then streams the farewell audio in 200 ms chunks at a natural speaking cadence, keeping the client's local buffer between 100 ms and 500 ms ahead of real-time playback. The final response.done event only arrives after the last chunk is dispatched (a median 436 ms later). Because the client only ever holds about half a second of audio at a time, a naive asyncio.sleep(1.0) hangs up a second into a goodbye that runs about 4 seconds: by then, only 1.35 to 1.59 s of it had reached the client in the probe calls. Reliable orchestration comes down to two explicit checkpoints: wait for session.updated before routing live frames upstream, and wait for response.done followed by speaker drainage before closing the socket. Our implementation builds directly on FrontDesk in agent.py , a base class providing booking lookups, a core listen() event loop, logging utilities, and barge-in handling. Call in call.py wraps this foundation to reinforce both edges. Prerequisites To run the project, ensure you have Python 3.13 and uv installed. You will also need a Dialog-RSN-1 API key from the API Keys tab in studio.poly.ai. The scripted test suite requires no hardware. If you plan to test call.py interactively, use headphones alongside your microphone to prevent local echo from feeding back into the agent. Set up the project Clone the companion repository and enter the project directory: git clone https://github.com/pmckeetx/barge-in-voice-agent.git cd barge-in-voice-agent Initialize your environment variables from the template: cp .env.example .env # Your Dialog-RSN-1 API key: sign in at https://studio.poly.ai and create one in the API Keys tab. POLY_API_KEY= # Optional: the full realtime endpoint. Defaults to wss://api.studio.poly.ai/v1/realtime. # DIALOG_REALTIME_URL= Add your API key to POLY_API_KEY . Leave DIALOG_REALTIME_URL commented out unless your key is for a different PolyAI host. Next, install dependencies and execute the offline mock suite: uv sync uv run pytest ... tests/test_agent.py ...... [ 28%] tests/test_audio.py .... [ 47%] tests/test_call.py ......... [ 90%] tests/test_live.py ss [100%] ======================== 19 passed, 2 skipped in 12.77s ======================== The offline tests use a simulated event server. To run the two skipped integration tests against live infrastructure, run POLYAI_LIVE=1 uv run pytest -m live . Note that this executes two real calls lasting around 70 seconds and consumes concurrent session quota. Step 1: Start listening at pickup The primary mistake in naive voice clients is tying hardware capture to network readiness. The sound card is local and ready immediately; the remote session takes multiple network round trips to establish. To prevent dropping words during that window, instantiate Call before opening the WebSocket, capturing all inbound audio into an in-memory ring buffer immediately: FRAME_S = 0.02 # the sound card and the scripted caller both hand over 20 ms frames BUFFER_CAP_S = 5.0 # the most of the caller's audio we hold while the session gets ready HANG_UP_CAP_S = 8.0 # the longest we wait for a goodbye to finish before hanging up anyway ... class Call(FrontDesk): ... def init(self, ws, speaker: Speaker, echo: EchoCanceller | None = None): super().init(ws, speaker, echo) self.held: deque[bytes] = deque(maxlen=round(BUFFER_CAP_S / FRAME_S)) # past the cap, the oldest frame goes self.live = False # True once the held audio has gone out and frames can go straight to the server self.goodbye: dict | None = None # the end_call call, while its goodbye is being spoken self.goodbye_spoken = asyncio.Event() self.hanging_up: asyncio.Task | None = None self.hung_up = asyncio.Event() async def send_audio(self, pcm: bytes): if self.hung_up.is_set(): return if self.live: await super().send_audio(pcm) else: self.held.append(pcm) The held queue stores 20 ms frames with a maximum size of 250 entries (5 seconds total). If the connection stalls for 10 seconds, the oldest frames drop off the back, so the agent hears the caller's last five seconds, not the stalest. Inside send_audio() , frames accumulate in held until self.live flips to True . Once live, frames route straight to FrontDesk.send_audio() , which encodes them into input_audio_buffer.append events. Once the call is marked as hung up, any trailing mic input is dropped safely. With the buffer ready, we invert our startup sequence: activate the local microphone stream first, and initiate the remote connection second. Here's call() in call.py : async def call(speakers: bool): """A live call on your microphone. The microphone opens first, then the connection.""" speaker = Speaker() echo = EchoCanceller() if speakers else None if e