I Built MoneyBuddy in 10 Days: A Multilingual AI Voice Agent for India's Financial Services🎙️

_What if getting help with a government financial scheme was as simple as having a conversation? That was the idea behind MoneyBuddy, the AI voice agent I built during 10 Days of Voice Agents - VoiceForBharat Edition. Over 10 days, MoneyBuddy evolved from a simple voice agent that could hear me and talk back into a system that can remember callers, use financial scheme data, make outbound calls, escalate difficult situations to humans, track call outcomes, and hand complex questions to a specialist agent. This post is the story of how I built it, what broke along the way, and what I learned. 🧩 The Problem Financial schemes and government programs can be difficult to navigate. There are eligibility conditions, documents, deadlines, different schemes, and a lot of information written for screens rather than conversations. For many users, especially users who are more comfortable speaking than typing, a voice interface can make that information easier to access. So I chose the Financial Services track. MoneyBuddy is designed as a conversational financial assistant that can help users understand Indian government schemes, perform basic eligibility-related checks, provide document information, remember useful context, and escalate situations where an AI should not make the decision itself. The goal isn't to replace a bank employee or government official. The goal is to make the first conversation easier. 🎙️ Why Voice? Voice changes the interaction completely. Instead of navigating menus such as: Scheme → Eligibility → Documents → Benefits → Apply the user can simply say: "Can you tell me if I might be eligible for PM Kisan?" The agent can ask questions conversationally and respond in the user's language. For an Indian audience, this also means handling conversations that aren't perfectly English. A user might speak Hindi, switch to English for a technical term, or use Hinglish naturally. MoneyBuddy was designed around that reality. 🏗️ How MoneyBuddy Works The architecture combines several components: ┌──────────────────────┐ │ User │ │ Voice / Browser / │ │ Phone Call │ └──────────┬───────────┘ │ │ Audio ▼ ┌──────────────────────┐ │ LiveKit │ │ Real-time transport │ └──────────┬───────────┘ │ ┌───────▼───────┐ │ Deepgram │ │ STT │ └───────┬───────┘ │ Text ▼ ┌──────────────────────┐ │ Gemini LLM │ │ Reasoning + Routing │ └───────┬───────┬──────┘ │ │ ┌──────────┘ └──────────┐ ▼ ▼ ┌───────────────┐ ┌─────────────────┐ │ Python Tools │ │ Scheme Specialist│ │ Memory / Data │ │ Agent │ └───────┬───────┘ └────────┬────────┘ │ │ └──────────────┬───────────────┘ ▼ ┌──────────────────┐ │ Murf Falcon │ │ TTS │ └────────┬─────────┘ │ Audio ▼ User The main pieces are: - LiveKit - real-time voice transport - Deepgram - speech-to-text - Google Gemini - language model and reasoning - Murf Falcon - text-to-speech - SQLite - persistent caller memory - Python function tools - financial scheme lookup and other actions - Next.js / React - frontend and analytics experience Murf describes Falcon as its lowest-latency TTS model for real-time conversations, with published benchmarks around 55 ms model latency and 130 ms time-to-first-audio. For MoneyBuddy, I used an Indian voice so the experience feels appropriate for the users I had in mind. 📅 What MoneyBuddy Learned I didn't try to build everything on day one. The system grew incrementally. Day 1 - Make It Talk The first milestone was simple: Hear the user → process speech → generate a response → speak it back. I chose the Financial Services track and configured an Indian Murf voice. At this stage, intelligence wasn't the goal. The goal was proving that the entire real-time voice loop worked. Day 2 - Give It a Job and Guardrails Once MoneyBuddy could talk, it needed a purpose. I defined three major objectives: - Help users understand financial schemes. - Improve basic financial awareness. - Keep users safe from common financial fraud risks. The guardrails became particularly important. MoneyBuddy must never ask users for things such as: - OTPs - PINs - passwords - CVVs - full card numbers - sensitive banking credentials It also must not promise that someone will definitely receive a loan or government benefit. This was one of my biggest lessons from the challenge: A voice agent isn't just an LLM that speaks. It needs boundaries. 🌐 Day 3 - Build the Interface The next step was making the experience feel like a product rather than a terminal application. The frontend shows states such as: - Ready - Connecting - Listening - Speaking - Call Ended It also includes a live transcript and microphone permission handling. For a voice interface, visual feedback matters. When there is no text box and no "send" button, the user needs another way to understand: Is the agent listening? Is it speaking? Is it still connecting? These states became part of the product experience. 🧠 Day 4 - Give It Memory A voice assistant that forgets everything after every call feels very limited. So I added persistent memory using SQLite. The agent can store useful caller information such as: - caller ID - name - language preference - relevant financial facts - interaction information The important part was not putting all of this directly into the prompt. Instead, MoneyBuddy has functions that allow it to look up and save information when appropriate. I also added a consent requirement before saving information. That distinction matters: Memory should be controlled by tools and permissions, not hidden prompt state. 🔎 Day 5 - Give It Real Financial Data Next, MoneyBuddy needed to do more than generate generic answers. I created a grounded financial scheme dataset containing information about schemes such as: - PM Kisan - PM Suraksha Bima Yojana - PM Jeevan Jyoti Bima Yojana - Atal Pension Yojana - PM Mudra Yojana The agent can call a function to retrieve scheme information instead of relying entirely on the model's memory. This also introduced another important voice-agent principle: When external data fails, the agent should say so. It should not silently invent an answer. 📞 Day 6 - Make MoneyBuddy Call the User This was one of the biggest changes. Instead of waiting for the user to open the browser and start a conversation, MoneyBuddy could make an outbound call. The use case I chose was: A government scheme deadline reminder for someone who had already been found eligible. Outbound calls introduce a completely different UX problem. The user didn't ask to call the agent. So the opening needs to immediately explain: - who is calling - why they're calling - how the user can stop the call That led to a very simple principle: Never make the user wonder who is speaking or why they are receiving the call. 🧑💼 Day 7 - Know When AI Should Stop An AI shouldn't try to solve every problem. For MoneyBuddy, I added escalation for situations where a human needs to take over, including possible fraud-related situations and problems that require a decision the AI cannot make. Before sharing information with a human, MoneyBuddy asks the caller for permission. The escalation summary contains only useful information: - who needs help - what happened - what was already checked - urgency - preferred language/follow-up method Sensitive information such as OTPs, PINs, passwords and account numbers should never be included. This was another major lesson: Knowing when not to answer is part of being a good voice agent. 📊 Day 8 - Measure the Calls Once the system became more complex, I needed a way to understand what was actually happening. So I built call analytics around real call data. The dashboard tracks metrics such as: - Total calls - Successful calls - Failed calls - Success rate - Call history - Call outcomes - Latency For MoneyBuddy, a successful call is tied to whether the user actually reaches the intended financial-service outcome rather than simply whether the call connected. My measured result Latency: [INSERT YOUR ACTUAL MEASURED LATENCY] Total calls: [INSERT ACTUAL NUMBER] Successful calls: [INSERT ACTUAL NUMBER] Failed calls: [INSERT ACTUAL NUMBER] Success rate: [INSERT ACTUAL %] I am deliberately using my measured values here rather than inventing benchmark numbers. 🤝 Day 9 - Add a Specialist The final major architecture change was moving from one agent to multiple agents. MoneyBuddy is the main financial assistant. But instead of making it an expert in everything, I created a dedicated: Government Scheme Specialist When the user asks something that requires deeper scheme-specific assistance, MoneyBuddy can hand the conversation over. The specialist has: - its own instructions - its own role - its own tools - its own voice MoneyBuddy uses Murf's Anisha voice, while the specialist uses Nikhil. The user doesn't need to repeat the entire problem. The conversation context moves with the handoff. One of the trickiest parts was getting the voice transition to happen correctly. I initially expected changing the agent instance to automatically change the active TTS pipeline. It didn't behave that way. The active TTS session had to be explicitly updated during the handoff. That became a good example of something I learned repeatedly during this challenge: Understanding the runtime architecture matters just as much as understanding the AI model. 🐛 The Problems I Actually Faced The project definitely did not work perfectly on the first attempt. Some of the most useful lessons came from debugging. 1. Multilingual speech isn't just an LLM problem Hindi speech could be understood, but getting the voice output to use the appropriate language/script behavior required careful STT, TTS, and prompt configuration. I had to pay attention to: - language detection - native scripts - TTS voice configuration - code-mixed speech - prompt instructions For example, Hindi should be generated in Devanagari rather than romanized Hindi when that is the intended output. 2. Voice handoffs aren't just model handoffs During Day 9, I discovered that changing the