How I Built Kisan Vaani: A Multi-Agent Voice AI Assistant for Indian Farmers

Over the last 10 days, I participated in the 10 Days of AI Voice Agents (Voice for Bharat Edition) challenge by Murf AI. My goal was to build a real-world voice application that solves an actual problem in India. I chose the Farm & Field Track and built Kisan Vaani (किसान वाणी) - an AI-powered voice assistant designed to help Indian farmers get instant market prices, weather updates, and agricultural doctor advice through natural Hindi voice calls. Here is a breakdown of what I built, the architecture behind it, the hardest bugs I ran into, and how you can run the project yourself. The Problem I Set Out to Solve In rural India, most agricultural information is locked behind text-heavy websites or mobile apps. For many farmers, navigating complex menus or reading long texts on smartphones is a major barrier. When a farmer needs to know today's mandi price for wheat in Noida, or needs urgent medicine advice for yellow rust disease on their crop, a 30-second voice call in native Devanagari Hindi is much more natural and effective than searching through an app. I designed Kisan Vaani to act like a personal phone call helper: - It greets farmers in natural Devanagari Hindi. - It looks up live Mandi commodity prices and weather forecasts. - It remembers previous caller profile information across calls using SQLite. - If the farmer reports a severe crop disease, it transfers the caller mid-session to a Specialist Crop Doctor Agent (Dr Samar), who speaks in a dedicated male doctor voice. System Architecture Here is how real-time audio and data flow through the application: flowchart TD Browser["Next.js Web Frontend & Audio Visualizer"] |WebRTC Stream| LiveKit["LiveKit WebRTC Cloud"] LiveKit STT["Deepgram Nova-3 Speech-to-Text"] LiveKit TTS1["Murf Falcon 2 TTS ('Anisha' Main Voice)"] LiveKit TTS2["Murf Falcon 2 TTS ('Samar' Doctor Voice)"] subgraph Agent Engine ["Python Agent Backend"] MainAgent["KisanVaaniMainAgent"] SpecAgent["CropDoctorSpecialistAgent"] LLM["Groq Llama 3.1 LLM"] end STT --> MainAgent MainAgent LLM MainAgent -->|Agent Transfer| SpecAgent MainAgent --> TTS1 SpecAgent --> TTS2 subgraph Storage ["SQLite Database"] DB1[("farmers table")] DB2[("escalations table")] DB3[("call_logs table")] end MainAgent Storage The Tech Stack - Voice Synthesis (TTS): Murf Falcon 2 (ultra-low latency <100ms TTFB) - Real-Time Transport: LiveKit Agents SDK & WebRTC Cloud - Speech Recognition (STT): Deepgram Nova-3 Multilingual - Language Model (LLM): Groq Llama 3.1 8B Instant - Database: SQLite (kisan_memory.db) - Frontend: Next.js 15, React, Tailwind CSS Key Features Built Over the 10 Days 1. Natural Hindi Voice with Ultra-Low Latency Using Murf Falcon 2 (Anisha voice), the agent responds almost instantly with a time-to-first-byte under 100 ms. The voice sounds warm and natural, making the conversation feel like talking to a real helper on the phone. 2. Caller Profile Memory (SQLite) Instead of asking for information repeatedly, Kisan Vaani remembers the farmer across calls. When Ramesh from Noida calls back, the agent loads his saved profile from the farmer's SQLite table and tailors its responses to his location and crop. 3. Live Mandi Prices & Weather Alerts I integrated real-time tools to fetch current market prices per quintal for crops like wheat, paddy, and mustard (via e-NAM price data), as well as district weather forecasts and rain probabilities (via the Open-Meteo API). 4. Human Escalation & Call Analytics Dashboard If a farmer faces an emergency or financial dispute, Kisan Vaani creates an escalation ticket in SQLite (escalations table) for a Krishi Vigyan Kendra (KVK) officer callback. All call outcomes, duration metrics, and success rates are rendered on a glassmorphism dashboard built with Next.js. 5. Multi-Agent Specialist Handoff When a farmer asks about crop diseases like yellow rust, the main agent (Anisha) announces a transfer: "I am connecting you with our senior crop doctor specialist." The session updates to CropDoctorSpecialistAgent, and the TTS voice dynamically switches to Murf Falcon Samar (a male doctor voice), who provides precise pesticide treatment remedies. Real Engineering Challenges & How I Solved Them Building real-time voice agents comes with unique edge cases. Here are the three main technical hurdles I worked through: 1. Stopping the LLM from Speaking Raw Tool Tags Early in testing, Llama 3.1 occasionally outputted text tags like {"name_or_id": "रमेश"} directly into the chat box. The Cause: Mentioning literal python function names inside the prompt text tricked the LLM into thinking it should output them as text instead of executing native background tools. The Fix: I removed all literal function names from the prompt text and added strict rules ordering the LLM to run background tools silently without outputting code tags. 2. Overcoming Groq 429 Rate Limits During Agent Transfers During multi-turn calls, Groq returned 429 Token-Per-Minute rate limit errors. The Cause: The accumulated conversation history sent to Groq grew too large on long calls. The Fix: When transferring the caller to the specialist doctor, I cleared the old chat context (session.chat_ctx.messages.clear()). This dropped the request payload back down to ~100 tokens, completely eliminating 429 rate limit errors. 3. Fixing LiveKit Read-Only Property Error Attempting session.tts = specialist_tts inside the handoff tool threw an AttributeError because session.tts is read-only in LiveKit Agents 1.4.5. The Fix: I passed the TTS instance directly into the agent class constructor: class CropDoctorSpecialistAgent(Agent): def init(self, specialist_tts, specialist_tools: list, farmer_name: str, district: str, crop_issue: str): super().init( instructions=get_crop_doctor_prompt(farmer_name, district, crop_issue), tts=specialist_tts, tools=specialist_tools, ) Now when session.update_agent(doctor_specialist) runs, LiveKit automatically updates both the agent logic and the Murf Falcon voice seamlessly. How to Set Up and Run Kisan Vaani If you want to run this project on your local machine, follow these steps: 1. Clone the Code git clone https://github.com/Aaddii17/murf-livekit-starter.git cd murf-livekit-starter 2. Install Dependencies Backend setup cd backend uv venv uv sync Frontend setup cd ../frontend npm install 3. Configure API Keys Create a .env.local file in both backend and frontend folders with your credentials: LIVEKIT_URL=wss://your-project.livekit.cloud LIVEKIT_API_KEY=your_livekit_key LIVEKIT_API_SECRET=your_livekit_secret MURF_API_KEY=your_murf_key DEEPGRAM_API_KEY=your_deepgram_key GROQ_API_KEY=your_groq_key 4. Launch the Dev Servers cd backend uv run python src/agent.py dev cd frontend npm run dev -- -p 3000 Open http://localhost:3000 in your browser to start talking with Kisan Vaani Conclusion & Code Links Building Kisan Vaani showed me how powerful combining WebRTC, fast TTS, and multi-agent LLM systems can be for solving accessibility problems in India. GitHub Repository: https://github.com/Aaddii17/murf-livekit-starter LiveKit Agents Framework: LiveKit Voice AI Docs Murf Falcon Documentation: Murf Falcon 2 Docs Thanks to Murf AI for hosting the #VoiceForBharat challenge! Top comments (0)