Building Arogya Seva: How I Built an Ultra-Low Latency Telehealth Voice AI for Bharat in 10 Days
Introduction: The Problem & The Mission Across rural and semi-urban India, access to timely healthcare advice is hindered by high patient-to-doctor ratios, geographical distances, and language barriers. Millions of citizens delay seeking medical advice simply because scheduling a clinic visit or understanding preliminary symptom urgency is daunting. Arogya Seva was created to bridge this gap as part of the #VoiceForBharat challenge (Track: Health Access). It is an empathetic, multilingual, real-time voice assistant designed to interact naturally in Indian English, Hindi (Devanagari script), and regional scripts. Why Voice? For millions of non-tech-savvy users or individuals in low-literacy regions, typing in an app or filling out complex forms is a friction point. Speaking directly over a phone call or web interface is the most accessible, natural, and human way to receive guidance. - System Architecture: How Audio & Data Flow To deliver a conversational voice experience, latency is paramount. A delay of more than 800ms between a user finishing their sentence and hearing a response breaks the illusion of natural conversation. The system is built on LiveKit Agents SDK with a modular pipeline: Speech-to-Text (STT): Deepgram Nova-3 transcribes spoken voice in real time. Brain (LLM): Google Gemini 2.0 Flash processes intent, applies clinical guardrails, and decides on function tool calls. Text-to-Speech (TTS): Murf Falcon (livekit-murf plugin, voice model en-IN-Anisha) streams ultra-low latency, human-like voice synthesis back to the user. Real-time Transport: LiveKit WebRTC (web frontend) and SIP Telephony (outbound/inbound phone calls). Memory & State: SQLite (agent_memory.db) for privacy-first caller persistence and escalation management. Mermaid diagram - Important Features Built Over 10 Days ๐ Feature 1: Sub-Second Voice Synthesis Powered by Murf Falcon Using Murf Falcon (en-IN-Anisha voice model), the agent achieves lightning-fast time-to-first-byte (TTFB). The voice sounds warm, empathetic, and natural-crucial for building trust with patients discussing health concerns. ๐ก๏ธ Feature 2: Strict Guardrails & Native Script Enforcement Health AI requires absolute safety. Arogya Seva follows strict operational boundaries: Red-Flag Clinical Emergency Protocol: Immediately flags chest pain, dyspnea, heavy bleeding, or acute trauma, urging callers to dial emergency 108. Native Script Enforcement: To ensure proper acoustic synthesis and avoid awkward transliteration, responses in Hindi are strictly produced in native Devanagari script (e.g., เคจเคฎเคธเฅเคคเฅ, เคเคช เคเฅเคธเฅ เคนเฅเค?), avoiding romanized "Hinglish". ๐ป Feature 3: Dynamic Frontend State & Audio Visualizers Built with Next.js and LiveKit Agents UI, the frontend displays real-time agent states: Listening (Visualized with dynamic frequency waveforms) Thinking (Tool execution state) Speaking (Fluid audio spectrum representation) ๐ง Feature 4: Privacy-First Memory with Explicit Consent Returning callers don't need to re-explain their location or age band. However, privacy is paramount: The agent explicitly asks: "May I save your name and basic health details so I can remember you next time?" Facts are stored only if explicit consent is given. Users can say "Forget me" at any time to wipe their records via forget_caller. ๐ ๏ธ Feature 5: Real-Domain Health Tools & Tool Chaining classify_symptom_triage: Categorizes symptoms into Self-Care / Low, Moderate / Consult Nurse, or High Urgent / Red-Flag. lookup_nearest_phc: Searches Primary Health Centres based on district. Tool Chaining: Automatically reuses district information saved in caller memory without re-asking the user. Graceful Failure: If the registry API is unreachable, the agent announces the offline status calmly and provides emergency helpline 104/108 numbers. ๐ Feature 6: Outbound Telephony & Mandatory Opt-Out For automated health reminders and follow-up calls: Two-Sentence Mandatory Opening: State WHO is calling, WHY, and HOW to opt out in the first two sentences. Instant Opt-Out: Saying "stop calling me" or pressing 9 immediately executes opt_out_caller in SQLite and terminates the call. ๐ Feature 7: Human Escalation & Reference IDs When situations exceed AI scope: Agent detects clinical doctor requests or red-flag symptoms. Agent requests explicit permission to create an escalation ticket. Upon agreement, create_escalation stores a sanitized summary (no passwords/PINs/Aadhaar) and returns a unique reference ID (e.g., ESC-8492). ๐ Feature 8: Call Analytics & Outcome Tracking Every call session logs structured metrics into SQLite, including call duration, triage classifications, escalation status, and resolution codes (triage_completed, phc_found, escalated, handed_off). ๐ Feature 9: Multi-Agent Specialist Handoff When callers request to schedule, modify, or cancel OPD appointments, the main agent invokes transfer_to_clinic_specialist: python @function_tool async def transfer_to_clinic_specialist(self, context: RunContext, reason: str) -> str: specialist = ClinicAppointmentSpecialist() context.session.update_agent(specialist) return "Handed off conversation to Clinic and Appointment Specialist." The session dynamically updates to ClinicAppointmentSpecialist, seamlessly swapping persona and toolsets without dropping the audio call! - Real Challenges & How We Solved Them Challenge 1: TTS Latency Spikes During Conversational Turns Problem: Default chunking caused 1.5-second pauses before the agent spoke. Root Cause: Large sentence buffers in the LLM-to-TTS pipeline. Solution: Integrated Murf Falcon with streaming tokenization and prewarmed Silero VAD models. This reduced speech synthesis latency to under 300ms! Challenge 2: Accidental Code-Mixed Script Bleed Problem: The LLM would occasionally respond to Hindi input with romanized Hindi ("Aapko kya takleef hai?"), causing the TTS to pronounce Hindi words with English phonetics. Solution: Implemented a mandatory system prompt guardrail enforcing native script generation (e.g., Devanagari for Hindi). Challenge 3: Agent Handoff State Management Problem: When handing off from general health to clinic specialist, tool contexts were losing call metadata. Solution: Leveraged context.session.update_agent(specialist) in LiveKit Agents SDK, ensuring the active WebRTC media room remained untouched while the prompt and function tools switched dynamically. - Practical Guide: Build & Run Your Own Voice Agent Want to build your own ultra-fast voice agent? Follow these steps! Step 1: Prerequisites Python 3.10+ & uv package manager Node.js 18+ & pnpm LiveKit Cloud account (URL, API Key, API Secret) Murf AI API Key (for Falcon TTS) Deepgram API Key (for STT) Google Gemini API Key (for LLM) Step 2: Clone & Configure Backend bash git clone https://github.com/viral-1998/VoiceOfBharat.git cd VoiceOfBharat/backend Create environment file (.env.local) cp .env.example .env.local Add your API keys to backend/.env.local: env LIVEKIT_URL=wss://your-livekit-project.livekit.cloud LIVEKIT_API_KEY=your_key LIVEKIT_API_SECRET=your_secret MURF_API_KEY=your_murf_key DEEPGRAM_API_KEY=your_deepgram_key GOOGLE_API_KEY=your_google_key Step 3: Run Backend Agent bash uv sync uv run python src/agent.py download-files # First time model download uv run python src/agent.py dev # Start live dev server Step 4: Run Frontend UI bash cd ../frontend pnpm install pnpm dev Open http://localhost:3000 in your browser, click Connect, and start speaking to your agent! - Code Spotlight: Specialist Handoff Logic Here is the exact Python implementation for handing off a LiveKit session from the main Telehealth assistant to the Appointment Specialist agent: python Function tool in Assistant class @function_tool async def transfer_to_clinic_specialist( self, context: RunContext, reason: str = "User requested appointment booking", ) -> str: """Transfer caller to Clinic & Appointment Specialist agent.""" specialist = ClinicAppointmentSpecialist() context.session.update_agent(specialist) call_id = getattr(getattr(context, "session", None), "call_id", "") if call_id: db.mark_call_success(call_id, outcome_summary=f"Handed off: {reason}") return "Handed off conversation to Clinic and Appointment Specialist." - What's Next? Future improvements for Arogya Seva include: Multi-lingual Voice Cloning: Adding localized voice accents across 10+ Indian regional languages using Murf Falcon's voice library. WhatsApp Telemetry Notifications: Sending automated SMS/WhatsApp appointment receipts following human escalations. EHR Integration: Connecting triage outcomes directly with ABDM (Ayushman Bharat Digital Mission) health IDs. - Links & Resources ๐ GitHub Repository: https://github.com/Viral-1998/VoiceOfBharat Repo โก Murf Falcon TTS Docs: Falcon 2 Documentation ๐๏ธ LiveKit Voice AI Quickstart: LiveKit Agents Guide ๐ Challenge Details: VoiceForBharat Challenge 2026 Thank you to Murf AI and LiveKit for hosting the 10 Days of AI Voice Agents (#VoiceForBharat Edition)! Top comments (0)
Comments
No comments yet. Start the discussion.