Building Arthashathi: My 10-Day Journey Building a Voice Agent for Bharat

From a Simple Voice Assistant to a Multi-Agent Financial Guide Over the last 10 days, I took part in 10 Days of Voice Agents - VoiceForBharat Edition, a challenge that pushed me to go beyond simply making an AI chatbot speak. My project is Arthashathi, a voice-first financial guide designed for Indian users. The idea was simple: What if accessing financial information could be as easy as having a conversation? Instead of navigating complicated websites, searching through government portals, or trying to understand financial terminology, a user can simply speak to Arthashathi and ask for help. Over these 10 days, Arthashathi evolved from a basic voice agent into a system with memory, tools, safety guardrails, government-scheme assistance, human escalation, outbound calls, analytics, and specialist-agent handoffs. This post is a summary of what I built, what went wrong, what I learned, and how someone else could start building a similar voice agent. 1. The Problem I Wanted to Solve India has a huge amount of financial and government-scheme information available online. The problem is not always the lack of information. The problem is accessibility. A user may have questions such as: - “Which government schemes am I eligible for?” - “What benefits does this scheme provide?” - “Is this UPI message a scam?” - “How can I stay safe from banking fraud?” - “Can someone explain this financial information to me simply?” For someone who is not comfortable navigating websites or typing long queries, a voice conversation can be much more natural. That is why I chose to build a voice-first financial assistant rather than another text-only chatbot. 2. Meet Arthashathi Arthashathi is an AI-powered voice financial guide designed primarily for Indian users. The main agent can help users with: - Government schemes - Banking safety - Fraud awareness - UPI safety - General financial guidance - Scheme eligibility - Human escalation when necessary The system is also designed to handle Indian languages and code-mixed conversations, allowing users to communicate more naturally instead of being forced to use formal English. The voice experience is powered by Murf Falcon, with real-time communication handled using LiveKit Agents. 3. How Arthashathi Works At a high level, the system follows a real-time voice pipeline: ┌──────────────────┐ │ USER │ │ Speaks naturally│ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ LiveKit │ │ Real-time Audio │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ Speech-to-Text │ │ (STT) │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ LLM │ │ Reasoning + │ │ Agent Decisions │ └────────┬─────────┘ │ ┌───────────┼────────────┐ ▼ ▼ ▼ Memory Tools Escalation │ │ │ └───────────┼────────────┘ │ ▼ ┌──────────────────┐ │ Specialist Agent │ │ when required │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ Text-to-Speech │ │ Murf Falcon │ └────────┬─────────┘ │ ▼ USER The important idea is that the LLM is not working alone. It is part of a larger system involving real-time audio transport, speech recognition, tools, memory, safety logic, and voice synthesis. 4. The Main Features I Built 🎙️ 1. Real-Time Voice Conversation The core of Arthashathi is a real-time voice interaction. The user speaks naturally, the system understands the request, generates a response, and speaks back. The voice layer uses Murf Falcon, allowing the project to focus on making the interaction feel conversational rather than robotic. 🧠 2. Memory for Returning Users One of the goals was to make Arthashathi more than a stateless chatbot. The system can maintain relevant user context so that returning users do not necessarily have to repeat everything from the beginning. This also introduced an important engineering challenge: correctly identifying the user across calls and ensuring that the correct memory was associated with the correct conversation. 🌐 3. Indian Languages and Code-Mixing Indian conversations are rarely limited to perfectly formal English. A user might naturally say something like: “Mujhe batao ei scheme-ta-r eligibility ki?” Arthashathi was designed with this reality in mind. The agent can handle Indian-language and code-mixed interactions while keeping important financial terminology understandable. 🛡️ 4. Safety Guardrails Financial assistance is a domain where an AI agent must know its limits. Arthashathi includes safety-oriented instructions around sensitive information and banking security. The system is designed not to request or expose sensitive credentials such as: - OTPs - PINs - CVVs - Passwords - Banking credentials The goal is not just to make the agent helpful, but to make it responsible. 🏛️ 5. Government Scheme Assistance Government schemes became one of the core use cases of Arthashathi. I created a dedicated scheme dataset containing information such as: - Scheme name - Benefits - Eligibility criteria - Relevant information dates The agent can use this information to help users understand schemes and determine whether a particular scheme may be relevant to them. 📞 6. Outbound Calls The project also explored the idea of proactive voice assistance. Instead of waiting for the user to call the agent, Arthashathi can be used for outbound reminders, such as informing eligible users about important scheme deadlines. This introduced additional considerations around consent, opt-out handling, and responsible calling. 🚨 7. Human Escalation An AI should not pretend it can solve every problem. For situations where human assistance is more appropriate, Arthashathi can escalate the conversation instead of attempting to answer beyond its capabilities. This was an important design principle throughout the project: Knowing when not to answer is also part of being a good AI assistant. 📊 8. Call Analytics I also added call-level analytics to understand what is happening during conversations. This allows the system to track useful information about calls and outcomes and provides a foundation for improving the agent based on real usage. 5. The Specialist Agent One of the most interesting parts of Day 9 was implementing agent handoff. The main agent should not try to be an expert at everything. So I created a dedicated: Government Scheme Specialist The architecture becomes: USER │ ▼ ┌───────────────┐ │ Arthashathi │ │ Main Agent │ └───────┬───────┘ │ General request? / \ YES NO │ │ ▼ ▼ Answer Specialist Handoff │ ▼ ┌─────────────────┐ │ Government │ │ Scheme │ │ Specialist │ └────────┬────────┘ │ ▼ USER The main agent first tells the user that it is connecting them to the specialist. The specialist then continues the conversation with the relevant context rather than forcing the user to repeat the entire problem. This was one of the most valuable lessons from the challenge: A good multi-agent system is not about having more agents. It is about giving each agent a clear responsibility. 6. The Problems I Faced This project definitely did not work perfectly on the first attempt. In fact, some of the most useful things I learned came from the failures. Problem 1: “Agent did not join the room” At one point, the frontend could connect to the application but the agent would not reliably join the LiveKit room. After debugging the system, one major issue turned out to be an agent-name mismatch. The frontend was requesting one agent name while the backend worker was registering under another name. Because the LiveKit dispatch system relies on the correct agent identity, the worker was not receiving the expected job. Fixing the naming mismatch and cleaning up duplicate workers resolved the dispatch problem. Lesson: When debugging real-time agent systems, don’t immediately assume the LLM or TTS is broken. Check the complete chain: Frontend ↓ Token ↓ LiveKit Room ↓ Agent Dispatch ↓ Worker ↓ Agent Session 7. The Latency Problem Another major challenge was latency. Initially, I tried making the system extremely aggressive about responding quickly. I reduced endpointing and silence thresholds so that the agent would react faster. That improved raw response speed - but introduced a new problem. The agent started interpreting normal pauses and breaths as the end of the user’s sentence. The result? The agent could interrupt too early, respond to incomplete sentences, or produce fragmented speech. I learned an important lesson: The lowest possible latency does not necessarily produce the best voice experience. A good voice agent needs a balance between: - STT endpointing - VAD silence detection - Interruption handling - LLM generation - TTS buffering - Natural conversational pauses Eventually, I moved toward a more balanced configuration instead of blindly minimizing every delay. 8. The Specialist Voice / Handoff Challenge Another interesting issue appeared during the specialist handoff. The specialist had its own Murf voice configuration, but changing the active agent did not automatically guarantee that the running audio pipeline would immediately use the specialist’s TTS configuration. This forced me to inspect how the LiveKit AgentSession manages its active TTS provider and how the handoff interacts with the running session. That experience taught me something important: Changing an agent’s logic and changing the active audio pipeline are not always the same operation. The handoff architecture therefore required careful testing of both the conversational state and the voice pipeline. 9. What I Learned The biggest lesson from these 10 days is that building voice AI is much more than connecting: Speech → LLM → Voice A production-oriented voice agent involves many moving pieces: Speech Recognition + LLM Reasoning + Tools + Memory + Safety + Real-Time Transport + TTS + Agent Routing + Analytics + Failure Handling A problem in any one of these layers can affect the entire user experience. I also learned that debugging a voice agent requires thinking about the whole system, not just individual files. 10. How You Can Build Your Own Voice Agent If you’re start