DEV Community

Let Me Talk to a Person: Handing Off a Voice Agent Call Without Making the Caller Repeat Themselves

When a caller says "let me talk to a person", a voice agent has one job left: get them to a person quickly, and make sure that person already knows why they're calling.

It sounds simple, but it's one of the places where voice agents most often annoy people. The agent asks the caller to explain the problem one more time, or tries to "help with that first", or transfers the call and the human answers with "Hi, how can I help you?" as if the last three minutes never happened.

The voice assistant I built for a dental course platform could hand off to an instructor, and the rule I'd keep for any agent like it is this: transfer on the first clear ask, and pass a short summary along so nobody has to repeat themselves. This post walks through how to do that on a Twilio phone line, with small TypeScript snippets for each part.

Treat the request as an action, not a topic

The first mistake is letting the LLM decide whether a handoff request is worth honoring. If the caller says "can I speak to someone?" and the model replies "Sure, but first, can you tell me more about the issue?", you've already lost that caller's trust. A person who asked for a human once and got pushed back will ask again, louder, and the rest of the call goes badly.

So I give the model a tool for it, something like transfer_to_human, and the prompt says plainly that when the caller asks for a person, the agent calls the tool right away.

I also don't rely on the model alone. A cheap check on the final transcript of each caller turn catches the common phrasings even if the model gets distracted:

const HANDOFF_PATTERNS = [
  /\b(talk|speak|transfer|connect)\b.*\b(human|person|someone|agent|representative|instructor)\b/i,
  /\breal person\b/i,
  /\boperator\b/i,
];

function wantsHuman(callerText: string): boolean {
  return HANDOFF_PATTERNS.some((re) => re.test(callerText));
}

This list will have false positives. "I already talked to someone yesterday" matches the first pattern. That's why the check doesn't transfer the call by itself. It flags the turn, and if the model didn't call the tool on a flagged turn, the agent asks one short question: "Do you want me to connect you to someone now?" A yes goes straight to the transfer. That one confirmation is fine. What isn't fine is a loop of clarifying questions before you let the caller go.

Make the transfer happen only once

Tool calls can repeat. The model might call transfer_to_human twice in one reply, or the regex flag and the tool can fire for the same turn. If each of those starts a new outbound dial, the human's phone rings twice, or the second update cancels the first one halfway through.

I keep the handoff as a small piece of call state and make the transfer function check it first:

type HandoffState = "none" | "requested" | "dialing" | "connected" | "failed";

async function startHandoff(call: CallSession) {
  if (call.handoff !== "none") return; // already in progress
  call.handoff = "requested";
  call.summary = buildSummary(call);
  await transferCall(call);
}

Because a single call is handled by one WebSocket connection on one server, a plain field on the session object is enough here. If your call state lives somewhere shared, like Redis, you'd want an atomic "set if still none" instead.

Moving the call off the media stream

If your agent runs on a bidirectional Media Stream, the call is executing <Connect><Stream> TwiML and your server is exchanging audio over a WebSocket. To hand the call to a human, you give the call new TwiML through the REST API. Twilio stops the current instructions, which ends the stream, and runs the new ones instead.

import twilio from "twilio";

const client = twilio(process.env.TWILIO_ACCOUNT_SID, process.env.TWILIO_AUTH_TOKEN);

async function transferCall(call: CallSession) {
  const vr = new twilio.twiml.VoiceResponse();
  vr.say("Okay, I'm connecting you to someone now.");
  const dial = vr.dial({
    timeout: 20,
    action: `${BASE_URL}/handoff-status?call=${call.id}`,
  });
  dial.number({ url: `${BASE_URL}/whisper?call=${call.id}` }, HUMAN_NUMBER);
  call.handoff = "dialing";
  await client.calls(call.callSid).update({ twiml: vr.toString() });
}

Two attributes do most of the work here. The url on <Number> is a "whisper": Twilio fetches that URL when the human picks up and plays the TwiML it returns to them only, before the two calls are bridged. The action on <Dial> is the URL Twilio requests after the dialed leg ends, and it includes a DialCallStatus parameter such as completed, busy, no-answer, failed or canceled. That's how you find out the transfer didn't work.

One small detail: say the "connecting you now" line in the new TwiML, not with your TTS right before the update. If you send TTS audio over the stream and then update the call, the stream can close before Twilio has played that audio, and the caller hears silence and then ringing.

The summary is the part people notice

The whisper endpoint returns a short spoken brief for the human:

app.post("/whisper", (req, res) => {
  const call = sessions.get(String(req.query.call));
  const vr = new twilio.twiml.VoiceResponse();
  vr.say(call?.summary ?? "Transferred call from the voice assistant.");
  res.type("text/xml").send(vr.toString());
});

What goes into that summary matters more than the plumbing. A good one answers three questions in a couple of sentences: who is calling, what they want, and what the agent already did. For a course platform that might be: "Sara, enrolled in the November batch. She's asking why her quiz score didn't save. I checked and the quiz shows as submitted with no score."

I build the summary mostly from structured call state rather than asking the model to retell the whole conversation. The caller's name, the course or account the agent looked up, and the tools it ran with their results are all facts the system already has. The model only writes the one line about what the caller wants. That keeps the summary short enough to hear in a few seconds, and it stops the model from inventing things that sound right but never happened.

This is also where the last post's fix pays off. If your conversation history only contains what the caller actually heard, the summary won't claim the agent told them something it never got to say.

A spoken whisper is easy to forget, so I'd also write the same summary somewhere the human can read it, such as the ticket, the CRM record or a message in the support channel, keyed by the caller's number.

When nobody picks up

Transfers fail. It's after hours, everyone is busy, or the call just isn't answered within the timeout. When the action URL reports anything other than a completed call, the worst thing you can do is drop the caller back into the bot with no explanation.

app.post("/handoff-status", (req, res) => {
  const call = sessions.get(String(req.query.call));
  const status = req.body.DialCallStatus;
  const vr = new twilio.twiml.VoiceResponse();
  if (status === "completed") {
    if (call) call.handoff = "connected";
    vr.hangup();
  } else {
    if (call) call.handoff = "failed";
    if (call) createCallbackTicket(call);
    vr.say("Sorry, nobody could take the call right now. I've passed your details on, and someone will call you back.");
  }
  res.type("text/xml").send(vr.toString());
});

Note that completed here means the human answered and the bridged call has since ended, so hanging up is right. For the failure case, be honest and specific. Don't promise a callback time you can't control, and don't send the caller back to the agent unless they ask for it. If you check your hours before dialing, you can skip the ringing entirely and go straight to the callback message.

Things I'd test

Most of this can be checked with a handful of test calls:

  • Ask for a human in the very first sentence of the call, before the agent knows anything. The summary should still say something useful, even if it's only "caller asked for a person right away".
  • Ask for a human in the middle of the agent's sentence, so the request arrives together with an interruption.
  • Say "I already spoke to someone yesterday" and check that it doesn't trigger a transfer without asking.
  • Make the human's phone not answer and listen to what the caller hears afterwards.
  • Ask twice in a row and make sure the human's phone rings once.

None of this needs a fancy model. The handoff is mostly about respecting the caller's request quickly and making sure the human starts the conversation already knowing what's going on.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.