Build an AI Receptionist With Whisper, Claude, Chatterbox and LiveKit: Architecture, Latency Budget and Starter Code

This post is AI-assisted.

Most "AI receptionist" tutorials stop at a chatbot that talks. A receptionist a dental clinic will actually pay for has to pick up a real phone line, understand messy speech, follow the clinic's rules, book into a real calendar and do all of it fast enough that the caller never notices a gap.

This is the architecture we use as a starting point, built from four pieces you can run yourself:

Why bother

Patient Prism's analysis of 11.5 million calls to 8,280 dental locations found 31% of callers hung up before reaching anyone (report). Only 42% of consumers say they leave a voicemail (CallRail survey).

Missed calls are lost bookings, which makes "answer every call" an easy thing to sell. The business math people quote is 100 clinics × $300 a month = $30,000 MRR. The engineering below is what makes that number survivable.

Architecture

PSTN caller │ SIP trunk ▼ LiveKit server (room per call) │ audio frames ▼ Agent worker (Python, LiveKit Agents) ├─ VAD + turn detector → decides "caller finished" ├─ STT adapter → faster-whisper on GPU (streaming chunks) ├─ LLM → Claude with tools: check_slots, book, take_message, transfer └─ TTS adapter → Chatterbox (clinic's consented voice), streamed sentence by sentence │ ▼ Clinic systems: calendar / practice software / CRM / SMS

Key design choices:

• One room per call. LiveKit treats a phone call like any other room participant, so the agent joins the room the SIP call lands in. Scaling is "more worker processes", not a rewrite.
• Stream everything. Do not wait for the full transcript, the full LLM answer or the full audio file. Start speaking on the first complete sentence.
• Tools, not prompts, for anything that changes state. The model never "says" it booked an appointment. It calls book_appointment, your code books it, and the result goes back to the model.

Latency budget

Callers start feeling awkward somewhere past a second of silence. Rough budget from "caller stops talking" to "agent audio starts":

These are targets to measure against, not guarantees. Measure each stage per call and log it. The biggest wins usually come from shorter prompts, streaming TTS by sentence and keeping the GPU workers warm.

Starter code shape

LiveKit's starter template (lk agent init my-agent --template agent-starter-python) gives you a working agent with hosted models. The API evolves quickly, so treat the snippet below as the shape of the code and check the current voice AI quickstart for exact signatures.

from livekit import agents
from livekit.agents import Agent, AgentServer, AgentSession, function_tool
from adapters import WhisperSTT, ClaudeLLM, ChatterboxTTS # your thin adapters

CLINIC_RULES = """You are the receptionist for Bright Smile Dental. Hours: Mon to Sat, 9am to 6pm. Book cleanings, checkups and consultations only. Never give medical advice. Transfer emergencies, billing disputes and upset callers to a human. Keep answers under two sentences."""

class Receptionist(Agent):
    def __init__(self):
        super().__init__(instructions=CLINIC_RULES)

    @function_tool
    async def check_slots(self, date: str) -> list[str]:
        """Return free appointment slots for a date (YYYY-MM-DD)."""
        return await calendar.free_slots(date)

    @function_tool
    async def book_appointment(self, name: str, phone: str, slot: str, reason: str) -> str:
        """Book a slot and send an SMS confirmation."""
        booking = await calendar.book(name, phone, slot, reason)
        await sms.send(phone, f"Confirmed: {slot} at Bright Smile Dental")
        return booking.id

server = AgentServer()

@server.rtc_session(agent_name="receptionist")
async def entrypoint(ctx: agents.JobContext):
    session = AgentSession(
        stt=WhisperSTT(model="large-v3", device="cuda"),
        llm=ClaudeLLM(),
        tts=ChatterboxTTS(voice_ref="voices/clinic_consented.wav"),
    )
    await session.start(room=ctx.room, agent=Receptionist())
    await session.generate_reply(instructions="Greet the caller and ask how you can help.")

if __name__ == "__main__":
    agents.cli.run_app(server)

The adapters are small classes that implement LiveKit's STT, LLM and TTS interfaces around faster-whisper, the Anthropic SDK and the Chatterbox Python package. Writing them yourself keeps every model swappable.

Production checklist

• SIP and numbers: connect a SIP trunk to LiveKit and port or forward the clinic's existing number, so patients dial what they already know.
• Barge-in: let callers interrupt. Stop TTS immediately when the caller speaks.
• Transfer path: a warm transfer to the front desk is a feature, not a failure.
• Voice consent: only clone voices you have written permission for. Chatterbox watermarks every output, which helps with provenance.
• Observability: store transcript, tool calls and per-stage latency for every call. Your support tickets will thank you.
• Data handling: health and personal details need careful storage and retention rules in many countries. Decide this before onboarding the first clinic.

Where the money actually is

The models are commodities now. The value is in the boring parts: integrations with each niche's software, reliable transfers, and reporting that shows the owner how many calls were saved. Pick one niche, nail those three, and the $30K MRR math stops being a slide.

Axionry is an AI product development studio that builds production voice agents. See the AI receptionist build, our voice AI development service, or price your setup with the voice AI cost calculator at axionry.com.