The caller heard silence for two seconds before the agent spoke

The Problem: Two Seconds of Dead Air

The bug report was one sentence: "callers keep talking over the greeting." I listened to the recordings and heard the same shape every time. The phone connects. The caller waits. One second of nothing. Two seconds of nothing. Then, right as the caller gives up and says "hello? are you there?", the agent starts its greeting, and now both of them are talking, and the whole call opens in a collision.

Nobody was talking over the agent to be rude. They were talking over it because they thought the line had dropped. Two seconds of dead air on a phone call is an eternity. People fill it.

I had a name for the number that was killing me before I knew its value: time to first audio byte. The gap between the caller finishing their turn and the first sample of the agent's voice actually reaching their ear. On this system it was around 2 seconds on the opening turn, and it felt every bit that long.

Timing the Pipeline Instead of Guessing

The first thing I did was stop theorizing and put timestamps on every stage boundary. Our pipeline was a straight line: speech-to-text produces a final transcript, the transcript goes to the LLM, the LLM produces a full completion, the completion goes to text-to-speech, TTS produces audio, audio goes out to the caller.

I logged a monotonic timestamp at each handoff and called in ten times. The averages, opening turn, cold:

• ASR final (from end of caller speech to committed transcript): about 300 ms
• LLM (request sent to full completion returned): about 1100 ms
• TTS (text sent to first audio chunk received): about 480 ms
• Network and playout queueing before the caller hears it: about 120 ms

That adds up to roughly 2 seconds, and the shape of it told the whole story. I was paying for the LLM to finish a complete sentence-and-a-half greeting before I sent a single character to TTS, and then paying for TTS to spin up a fresh connection before it produced its first chunk. Every stage waited politely for the previous stage to be completely done. It was a relay race where each runner insisted on crossing the finish line before handing off the baton.

The Waits I Was Paying For That I Did Not Need

Three of those waits were avoidable, and none of them required a faster model or a faster voice.

Wait one: buffering the whole LLM completion. I was calling the LLM, awaiting the full string, and only then handing it to TTS. But the greeting's first sentence exists long before the last sentence. If I stream tokens out of the LLM and start synthesizing the moment I have a complete sentence, TTS can begin speaking while the LLM is still writing.

Wait two: TTS cold start. The synthesizer opened a fresh streaming connection on every turn. That handshake was a big chunk of the 480 ms. A connection held open and warmed, ready before the caller even finishes talking, drops first-chunk latency hard.

Wait three: no acknowledgement. Even with everything above, there is an irreducible floor. For that last stretch I stopped trying to make the agent faster and made it sound present instead. A short spoken acknowledgement (a warm "mm-hm, let me pull that up") emitted immediately covers the remaining gap while the real answer synthesizes behind it.

Streaming the LLM into TTS at Sentence Boundaries

The core change was replacing "await the whole completion, then speak" with "speak each sentence as it finishes." I buffer streamed tokens, watch for a sentence-ending boundary, and flush that sentence to the streaming TTS the instant it is complete. TTS starts producing audio off the first sentence while the LLM is still generating the second.

Here is the piece that matters, the boundary-aware bridge between the two streams:

import asyncio
import re

# Split on sentence-final punctuation followed by a space or end of chunk.
_BOUNDARY = re.compile(r"(.+?[.!?])(\s+|$)", re.DOTALL)

async def stream_llm_to_tts(llm_stream, tts):
    """Forward LLM tokens to a streaming TTS one sentence at a time.
    llm_stream yields text deltas.
    tts.speak(text) accepts partial text and streams synthesized audio out on its own;
    tts.finish() closes the utterance.
    """
    buffer = ""
    first_audio_at = None

    async for delta in llm_stream:  # e.g. "Sure" ", I can " "help. What..."
        buffer += delta

        # Emit every complete sentence sitting in the buffer right now.
        while True:
            match = _BOUNDARY.match(buffer)
            if not match:
                break
            sentence = match.group(1).strip()
            buffer = buffer[match.end():]
            if sentence:
                if first_audio_at is None:
                    first_audio_at = asyncio.get_event_loop().time()
                await tts.speak(sentence)  # begins synthesizing immediately

    # Flush the trailing fragment (no terminal punctuation on the last bit).
    tail = buffer.strip()
    if tail:
        await tts.speak(tail)
    await tts.finish()
    return first_audio_at

The detail that earns its keep is flushing on the first boundary, not waiting for a comfortable buffer of two or three sentences. The greeting "Thanks for calling, this is the support line." becomes speakable audio the moment that first period lands, which on a streamed completion is a couple hundred milliseconds in, not eleven hundred.

The Filler Token That Bought the Rest

Sentence streaming took the LLM contribution to first audio from about 1100 ms down to about 250 ms. Warming the TTS connection took its first-chunk latency from about 480 ms to about 90 ms.

Good, but the opening turn still had the ASR-final wait in front of everything, and on a slow LLM first token you can still feel a beat. So I stopped trying to win the race and cheated the perception instead. The instant ASR commits a final transcript, before the LLM has produced a single token, I send one short pre-synthesized acknowledgement to the caller:

async def handle_turn(session, transcript):
    # Fire an instant acknowledgement so the line never sounds dead.
    # This audio is pre-warmed and starts playing in well under 100 ms.
    await session.tts.speak_cached("ack_soft")  # "mm-hm, one sec"

    llm_stream = session.llm.stream(transcript)  # real answer, in parallel
    first_audio_at = await stream_llm_to_tts(llm_stream, session.tts)
    return first_audio_at

The acknowledgement is not filler in the pejorative sense. A human agent does exactly this. You say "sure, let me look" the instant you understand the question, and the customer relaxes, because the sound told them they were heard. The caller now hears something within about 150 ms of finishing their sentence. The real answer arrives underneath it, and the two stitch together into one continuous turn.

What Shipped, and What I'd Tell Past Me

What went to production: token-level streaming from the LLM into a streaming TTS with a flush on every sentence boundary, a TTS connection warmed and held open before the caller finishes speaking, and an immediate cached acknowledgement fired on ASR-final so the line is never silent while the real answer synthesizes.

Measured time to first audio on the opening turn went from about 2 seconds to about 150 ms of acknowledgement plus roughly 400 ms to the substance. Nobody talks over the greeting anymore, because there is no dead air to fill.

If I could send one note back to the version of me who wired up that first pipeline: stop treating latency as one number to shave down. It is a budget split across four stages, and the biggest line item was almost never where I assumed. I would have bet the money was in the model. It was in the fact that I made every stage wait for the previous stage to finish completely before it started. Streaming changed that, and on this pipeline it did most of the work. The moment the first sentence exists, speak it.

The second note is about those opening 200 ms. What the caller needs there is not a correct answer, it is any sound at all. Silence on a phone reads as a dropped call, and a caller who thinks the call dropped starts talking, and then you are debugging a collision you created by being quiet. So I say something small and instant, and let the real answer arrive underneath it. On the opening beat, being early mattered more than being right.