Building RupeeGPT: A Multilingual Voice AI Financial Assistant for Bharat

How I Built RupeeGPT: A Voice-First AI Financial Assistant for India in 10 Days #VoiceForBharat | Built with the fastest TTS API - Murf Falcon | 10 Days of Voice Agents Ten days ago, I started with a blank repo and a challenge: build a production-ready voice AI agent for Indian users - one that could speak naturally in English, Hindi, and Hinglish; remember returning callers; escalate to humans when things got serious; and hand off conversations to specialist agents without ever making the caller repeat themselves. What came out the other side is RupeeGPT - a conversational AI financial assistant that helps any Indian user navigate banking, UPI, government welfare schemes, loans, and financial safety. Here's everything I built, what broke, how I fixed it, and how you can build your own. The Problem: Finance Advice Is Inaccessible to Most Indians India has over 500 million smartphone users, but financial literacy remains a barrier for hundreds of millions of people - especially in tier-2 and tier-3 cities and rural areas. The information exists: government scheme portals, RBI guidelines, banking apps. But it's buried in bureaucratic language, English-only interfaces, and long PDF documents. A voice agent changes that. You don't need to read anything. You don't need to know the right portal URL. You just talk. Who it's for: First-generation bank account holders, rural farmers checking PM Kisan eligibility, street vendors exploring PM SVANidhi loans, anyone who's ever been told to "read the fine print" and couldn't. Why voice: Voice meets people where they are. It removes the literacy barrier, it's faster than navigating apps, and for many rural users, calling is the most intuitive interface they know. The Architecture 🎙️ User speaks → Deepgram STT (nova-3, multilingual) → Gemini LLM (gemini-3.5-flash-lite via LiveKit Inference) → Murf Falcon TTS (Anisha - Indian English, en-IN) → LiveKit real-time transport → 🔊 User hears The stack: - Backend: Python 3.12, LiveKit Agents SDK, uv for dependency management - Frontend: Next.js 14 (App Router), TypeScript, Tailwind - Memory: MongoDB Atlas (persistent caller profiles) - Transport: LiveKit (WebRTC) - TTS: Murf Falcon - more on why this matters below Feature 1: An Indian Voice That Actually Sounds Indian The default for most TTS-backed voice agents is a US English voice. For an Indian user asking about PM Kisan Samman Nidhi, hearing a generic American accent reading scheme names in English phonetics feels jarring and impersonal. Murf Falcon's Anisha voice - Indian English, en-IN , Conversation style - changes this completely. But there was a subtlety: even with an Indian voice, scheme names like "PM Kisan Samman Nidhi" or "Pradhan Mantri Jan Dhan Yojana" are read with English phonetics when spelled in Roman script. My fix: a TTS pronunciation layer (tts_hindi.py ) _ENGLISH_TO_HINDI: tuple[tuple[str, str], ...] = ( ("pm kisan samman nidhi", "पीएम किसान सम्मान निधि"), ("pm jan dhan yojana", "पीएम जन धन योजना"), ("pradhan mantri jan dhan yojana", "प्रधानमंत्री जन धन योजना"), ("pm svanidhi", "पीएम स्वनिधि"), ("aadhaar", "आधार"), ("yojana", "योजना"), # ... more ) Before any text reaches Murf Falcon, it passes through this whitelist rewriter. Known Hindi/Indian terms are converted to Devanagari, so the voice says "पीएम किसान सम्मान निधि" - exactly as a native speaker would say it on TV - instead of "P M Kisan Samman Nidhi" with English stress patterns. The rewriter is safe to apply for every language mode: a pure-English sentence with none of these terms passes through byte-for-byte unchanged. I also built a detect_language() function that classifies each user utterance as english , hindi , or hinglish using Devanagari character detection and a curated Hinglish marker word list: HINGLISH_MARKERS = ("mujhe", "kaise", "kya", "chahiye", "baat", "namaste", "yojana", "sarkari", "paise", "rupaye", "bharat", ...) The agent mirrors the caller's language - answers in Hindi if they speak Hindi, Hinglish if they code-switch - without ever asking them to repeat. Feature 2: Personality, Objectives, and Safety Guardrails The system prompt defines the entire character of RupeeGPT: what it will help with, what it refuses, and how it escalates. Key guardrails baked into the system prompt: - Never ask for OTPs, PINs, passwords, or Aadhaar/PAN numbers - ever, for any reason - Never guarantee loan approval, scheme eligibility, or returns - Never impersonate bank officials or government employees - Two mandatory escalation triggers: suspected fraud/unauthorized transactions, and official decision overrides (e.g., custom loan limit requests) For the two escalation scenarios, the agent must: - Stop assisting and explain the situation - Name exactly what information it will share - Get explicit spoken consent before proceeding - Call create_escalation() only after consent This pattern - ask before acting, require a clear YES - became a design principle throughout the whole project. Feature 3: Multilingual TTS - English, Hindi, and Hinglish The TTS node hooks into the LiveKit Agents pipeline using Agent.default.tts_node : async def tts_node(self, text, model_settings): language = self._tts_language() async def _tracked(): async for part in tts_hindi.stream_for_tts(text, language=language): yield part async for frame in Agent.default.tts_node(self, _tracked(), model_settings): yield frame The stream_for_tts function accumulates the LLM's streaming text output into complete sentences before passing each sentence through the Devanagari rewriter. This is important: if a scheme name like "PM Kisan Samman Nidhi" were split across two streamed chunks, the phrase-level rewriter would miss it. Feature 4: Persistent Memory for Returning Callers Every caller gets a persistent browser ID (stored in localStorage and passed as a LiveKit participant attribute). The agent reads this at the start of every session and calls lookup_user() to fetch any saved profile from MongoDB. But here's the part that took the most iteration: consent architecture. The agent is not allowed to save any personal fact without: - The caller explicitly sharing the fact - The agent asking whether to remember it (naming the exact fact) - The caller saying a clear YES - The agent calling grant_user_memory_consent() with that exact value - Only then calling save_user_memory() with the same value # Tools must be called in sequence, only after explicit spoken consent: # 1. grant_user_memory_consent(name="Rahul", ...) # 2. save_user_memory(name="Rahul", ...) The save_user_memory tool actively checks the in-session consent store and blocks saves for anything that wasn't consented to in the current call. Returning callers are greeted naturally: "Namaste Rahul, welcome back. Would you like to continue from PM Jan Dhan Yojana?" The MongoDB document looks like this: { "user_id": "abc123", "name": "Rahul", "language_preference": "Hinglish", "facts": { "schemes_checked": ["PM Jan Dhan Yojana"], "eligibility_answers": { "income_bracket": "below 3 lakh", "farmer": true } }, "last_interaction": "2026-08-14T10:30:00Z" } Feature 5: Tools That Fetch Real Data Three function-calling tools give the agent live (or near-live) data: find_eligible_schemes - Matches the caller's profile (age, state, income, occupation, caste, residence, disability, BPL status) against a local dataset of Indian government welfare schemes. Returns preliminary matches with names, benefits, documents required, and official portal URLs. get_usd_inr_rate - Live USD/INR exchange rate. get_lending_rates - Current base lending rates and MCLR data. The scheme-matching tool uses a careful LLM prompt to avoid hallucination: - It never invents schemes or eligibility criteria - If the result set is empty, it says so clearly - If the tool errors, it says only that it cannot check right now - never speculates Feature 6: Outbound Phone Calls Using LiveKit's SIP integration, the agent can place outbound calls to real phone numbers. The session pipeline automatically detects SIP participants and switches the noise cancellation model: noise_cancellation=lambda params: ( noise_cancellation.BVCTelephony() if params.participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP else noise_cancellation.BVC() ) BVCTelephony is optimized for the narrowband audio characteristics of SIP/PSTN calls. The call analytics dashboard logs whether each session was a web or sip call. Feature 7: Human Escalation with a Live Dashboard When a caller reports suspected fraud or requests a decision override, the agent collects their name, contact number, issue summary, and urgency level - all with explicit consent - then calls create_escalation() . This writes to escalations.json (read by the Next.js frontend) and POSTs to a webhook endpoint. The Escalation Desk (/demo route) shows open escalations in real time: - Caller name, contact, issue summary, urgency badge - Status: Open / In Progress / Resolved - Reference ID (e.g., ESC-492716 ) that the agent reads back to the caller Feature 8: Call Analytics Dashboard Every call session - web or SIP - is logged on close. The _on_close handler fires when LiveKit closes the room: def _on_close(ev) -> None: call_record = { "id": ctx.room.name, "created_at": start_time, "ended_at": datetime.now(timezone.utc).isoformat(), "duration_seconds": round(duration, 2), "success": userdata.get("success", False), "success_reason": userdata.get("success_reason", ""), "call_type": call_type, # 'web' or 'sip' "user_id": userdata.get("user_id", "") } # Write to calls.json + POST to Next.js API A call is marked successful when the caller either checks their government scheme eligibility or creates a human escalation. The /dashboard page shows: - Total calls, success rate, successful calls, failed calls (live-updating every 3s) - Filterable call log table with duration, type badge, timestamp, and outcome Feature 9: Agent Handoff to a Specialist This was Day 9, and probably the most elegant feature tec