AI Agent Answers the Phone and Hands You to a Real Person

The Cast

Everybody's seen the demo: you call a number, an AI picks up, and you can suggest to hand over to a real person. It answers a real phone number, talks back in whatever language you speak to it, looks up answers to your questions, and hands you off to a real person when you ask.

Picture a few characters and a middleman:

• The caller: a human on a phone.
• The phone system: the thing that actually answers the call. (We used FreeSWITCH, a free, open-source phone server.)
• The AI: a voice model you can talk to and it talks back. Not text. Voice in, voice out.
• The middleman: a small program we wrote that sits in between, passing audio back and forth and deciding what to do next.

You ──▶ Phone system ──▶ Our middleman ──▶ The AI
       📞 (FreeSWITCH)    (our code)        🧠
        ▲                                   │
        └──────── "transfer" ◀──────────────┘
        and: 📚 look up an answer

Why the Middleman?

Couldn't the phone system just talk to the AI directly? Not really, and the reasons are basically the whole reason this project exists:

• Secrets: the passwords and keys live safely in our code, never on the phone box.
• Actions: when the AI decides "I should look that up" or "I should transfer this call," someone has to actually do it. The phone system has no idea how. Our middleman does.
• Control: one place to log what happened, cap how long calls run, and swap the AI out later.

That's the shape. Now the interesting failures.

1: "Voice in, voice out" is newer and better than it sounds

The old way to make a phone bot was a three-step relay race: turn the caller's speech into text, send the text to an AI, turn the AI's text back into speech. Three handoffs, and the meaning got a little more bruised at each one.

The new voice models do all three in one. You hand them audio, they hand you audio. And two lovely things fell out of that for free:

• It's faster. One step instead of three. On a phone call, the gap between "you stop talking" and "it starts talking" is everything.
• It just speaks your language. Someone called and spoke Tagalog, and it answered in Tagalog. We wrote zero code for that. No "press 1 for English."

2: The audio has to arrive like a heartbeat

This is the one that ate my time. Phone audio isn't a file you send all at once. The phone line expects that rhythm and nothing else. Our AI, though, doesn't talk in a steady drip. It thinks for a beat and then hands you a big gulp of audio all at once - two seconds of speech in a single burst.

If you just shove that gulp at the phone line as fast as it arrives, the line chokes. It has a tiny bucket (about a tenth of a second) and the rest spills on the floor. The caller hears the AI stutter and drop words.

The fix is almost embarrassingly simple once you see it: catch the gulp in a queue, then release one small slice every 20 milliseconds, matching the heartbeat the phone line wants.

// Catch the AI's audio, then feed the phone line ONE small frame every 20ms.
const queue = [];
setInterval(() => {
  const frame = queue.shift();
  if (frame) phoneLine.send(frame); // one heartbeat's worth of sound
}, 20);

That little 20 is the difference between "sounds like a person" and "sounds like a robot losing signal." The lesson that generalizes: when something plays in real time, sending it as fast as you can is a bug. You send it as fast as it's meant to be heard.

(There's a second gremlin in the same neighborhood: the AI speaks at one audio "resolution" and the phone line runs at a lower one, so you also have to gently down-convert every slice - the audio equivalent of resizing a photo without making it blocky. Same idea, less drama.)

3: Let people interrupt and be honest about the limits

Real conversations have interruptions. You start answering, the other person jumps in, and you stop. A bot that plows through its whole sentence while you're talking feels like a hold-menu, not a helper.

The good news: the AI tells us the instant it hears you cut in. The moment it does, we dump everything we were about to say and go quiet.

The humbling news: you can only un-say the words you haven't handed over yet. Anything already dripping through the phone line will finish playing - you can't reach into the wire and grab it back.

So the trick isn't a clever cancel button; it's never getting too far ahead of yourself. We hand the phone line as little audio as possible at a time, so when you interrupt, there's barely anything left in flight to talk over you.

The takeaway I keep coming back to: how fast you can stop is decided by how much you've already committed. That's true of a lot more than phone calls.

4: Don't hang up on people (a surprisingly deep rule)

Handing the caller to a real human sounds trivial. It's where we found the most sharp edges.

Say goodbye before you leave. The AI wants to say "Connecting you to billing, one moment" - but its decision to transfer arrives a beat before those words do. If you transfer the instant it decides, you cut the caller off mid-sentence. So we hold the transfer, let the sentence finish playing, and then move the call. Decide now, act after the goodbye.

Unhook yourself first. Our middleman is quietly listening to the call. If we transfer without unhooking, we'd follow the caller right into their private conversation with the human. Not okay. So: stop listening, then transfer.

If the handoff fails, stay. The worst possible outcome is dropping someone into dead air. So if a transfer can't complete for any reason, we keep the caller connected and let the AI apologize and offer something else - rather than dumping them and vanishing.

If one line of code captures the whole spirit, it's this:

transfer(...).then((result) => {
  if (result.error) return; // handoff failed → DON'T hang up; keep them with us
  endOurSide(); // only leave once they're safely with a human
});

Fail toward the human, never toward the dial tone. That's the rule.

5: How it actually answers a question

When you ask "what time is breakfast?", the AI doesn't secretly know your business. Here's the honest sequence:

1. You ask.
2. The AI realizes it needs facts and basically raises its hand: "go look this up for me."
3. Our middleman searches a knowledge base (a stack of your real answers) and hands back the best matches.
4. The AI reads those and speaks a grounded answer - in your language.

The AI decides when to look something up; we own how good the lookup is. The one trap worth naming: if the search is even slightly misconfigured, it doesn't throw an error - it quietly returns confident nonsense, and the AI will happily say it out loud. So "the answers are subtly wrong" is a config bug, not an AI bug. Check the plumbing before you blame the model.

6: A forgotten call is a leaking faucet

The AI charges for every second it's awake on a call. A caller who goes silent and wanders off is a slow money leak. So two humble timers earn their keep every single day:

• Quiet too long? (say, 30 seconds of silence) - wrap up gently and hang up.
• On too long, period? - a hard ceiling so no call can run forever.

Not glamorous. Absolutely essential. Put them in on day one, not after the first surprising invoice.

Built on FreeSWITCH (a free phone server), a small Node/TypeScript middleman, a voice-to-voice AI model, and a searchable knowledge base. Every hard part lived in the middleman - and none of it was really about the AI.