Ava: Intelligent AI Voice Assistant for English Learning

Building Ava: A Real-Time AI Voice English Learning Assistant with Murf Falcon & LiveKit 10 Days of Voice Agents - VoiceForBharat Edition Over the past 10 days, I built Ava, an AI-powered voice English learning assistant designed to help learners practice English through natural, real-time conversations. The project started as a simple voice agent and gradually evolved into a complete voice AI system with memory, live tools, outbound calling, human escalation, call analytics, and specialist-agent handoffs. The biggest goal was not just to make an AI that can talk, but to build an agent that can listen, understand, remember, take actions, and know when it needs help from a human or another specialist agent. What is Ava? Ava is a conversational AI voice assistant designed primarily for English-learning practice. A learner can talk to Ava naturally instead of typing messages. Ava listens to the learner's voice, understands the conversation, generates a response, and speaks it back in real time. The core voice pipeline is: 🎤 User Voice → Deepgram STT → Gemini LLM → Murf Falcon TTS → 🔊 User The real-time communication layer is handled using LiveKit. 💡 Why Voice? Learning English is not only about reading and writing. Speaking and listening are equally important. A voice-based assistant makes it possible for learners to practice: 🗣️ Speaking English 👂 Listening comprehension 💬 Natural conversations 📚 Vocabulary ✍️ Common mistakes 🎯 Confidence while speaking Instead of interacting with a chatbot through text, learners can have a more natural conversation with Ava. Day 1 - Voice Agent Foundation 🎤 I started by setting up the basic voice agent using LiveKit. The initial goal was simple: Listen to the user → process the speech → generate an answer → speak the answer. This created the foundation for the rest of the project. Day 2 - Personality & Guardrails 🧠 I gave Ava a clear personality and behavior. Ava was designed to be: Friendly Short and conversational Respectful Helpful Focused on English learning I also added safety guardrails so Ava doesn't assist with things like hacking, fraud, password theft, OTPs, or other unsafe requests. Day 3 - Frontend Experience 💻 I customized the frontend to make the voice-agent experience easier to understand. The interface provides feedback about the current state of the voice conversation and connects the user with the AI agent. This helped turn the project from a backend experiment into an actual user-facing application. Day 4 - Memory 🧠💾 One of the most interesting features was adding persistent memory. Ava can remember useful information about returning users, such as: Name Language preference Current learning level Topics covered Common mistakes But there is an important privacy rule: Ava asks for permission before saving useful personal information. This made the memory system more user-controlled instead of automatically storing everything. Day 5 - Live Tools 🌦️ I added a live weather tool. For example, a user can ask: "What's the weather in Bhopal?" Instead of guessing, Ava calls a weather API, retrieves current information, and explains the result naturally. This was an important step because the agent could now take an action instead of only generating text. Day 6 - Outbound Voice Calls 📞 Next, I extended the project to support outbound calling. The agent could initiate a call and interact with a learner through a phone/SIP-based workflow. I also added rules for outbound conversations: Identify itself as an AI Explain why it is calling Ask whether the learner has time Respect "stop" requests End the call if the learner is busy Never pressure the learner This introduced another important lesson: Voice agents need to respect the user's time and consent. Day 7 - Human Escalation 👨🏫 Ava doesn't try to solve everything. If a learner becomes confused, frustrated, or explicitly asks for a human teacher, Ava can offer to escalate the issue. Before creating an escalation request, Ava asks for permission. The request can contain information such as: User name Problem What Ava already tried Urgency Language Preferred follow-up method A reference ID is then generated for the request. Day 8 - Call Analytics 📊 I built a call analytics system to track conversations. The dashboard can provide information such as: Total calls Successful calls Failed calls Pending calls Success rate Call duration This helped me understand that building a voice agent isn't only about the conversation itself. Observability and analytics are also important parts of a real voice AI application. Day 9 - Specialist Agent Handoff 🔄 I then added a specialist-agent architecture. Ava acts as the main assistant. When a user asks something related to call analytics, Ava can transfer the conversation to a Call Analytics Specialist. For example: User: "How many calls were successful?" Ava can respond: "I'll connect you to our call analytics specialist." The specialist then takes over the conversation. This introduced the idea of multi-agent collaboration. Day 10 - Share & Reflect 🚀 The final day was about documenting the project, sharing what I learned, and making the project useful for other developers. The biggest lesson from these 10 days was: A voice agent is much more than an LLM that can speak. It needs speech recognition, reasoning, voice synthesis, real-time communication, memory, tools, safety, observability, and reliable handoffs. 🏗️ Architecture The overall system looks like this: ┌──────────────────┐ │ User │ │ Voice / Browser │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ LiveKit │ │ Real-time Audio │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ Deepgram │ │ STT │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ Gemini LLM │ │ Reasoning / Chat │ └────────┬─────────┘ │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ ┌──────────┐ ┌──────────┐ ┌────────────┐ │ Memory │ │ Tools │ │ Specialist │ │ Database │ │ Weather │ │ Agent │ └──────────┘ └──────────┘ └────────────┘ │ ▼ ┌──────────────────┐ │ Murf Falcon │ │ TTS │ └────────┬─────────┘ │ ▼ 🔊 Voice Response ⚙️ Tech Stack AI & Voice Murf Falcon - Text-to-Speech Deepgram - Speech-to-Text Google Gemini - LLM Silero VAD - Voice Activity Detection Real-Time LiveKit LiveKit Agents Backend Python SQLite HTTPX Flask Frontend Next.js React TypeScript 😅 Challenges I Faced The project definitely did not work perfectly on the first attempt. I encountered several real-world problems during development: 🔐 GitHub Secret Scanning GitHub blocked one of my pushes because an API key was detected inside a commit. I learned that simply changing the current file isn't enough if the secret still exists in Git history. The solution was to remove the secret from the affected commits/history and keep only safe placeholders such as: GOOGLE_API_KEY=your_google_api_key 🌐 LiveKit Connection Problems I also encountered LiveKit connection problems such as: RegionError("region fetch timed out") and: signal connection timed out At one point, DNS resolution itself was failing. Testing with commands such as: nslookup murf-voice-agent-g91x9izl.livekit.cloud curl -I https://murf-voice-agent-g91x9izl.livekit.cloud helped identify whether the problem was in my application or the network connection. 🎙️ Deepgram Connection Timeout Another issue was: Deepgram did not receive audio data or a text message within the timeout window This taught me that real-time voice systems are highly dependent on stable network connections and continuous audio flow. 🔄 Voice Interruption & Latency I also worked through issues where Ava's voice would break or conversations would feel less smooth. This required looking at the complete pipeline: Microphone → LiveKit → STT → LLM → TTS → LiveKit → Speaker rather than assuming the problem was only in the TTS. 🔐 Security One of the most important lessons was never exposing API keys. Keys should stay inside environment variables: .env.local and should never be committed to GitHub. The repository should contain only safe examples: GOOGLE_API_KEY=your_google_api_key MURF_API_KEY=your_murf_api_key DEEPGRAM_API_KEY=your_deepgram_api_key LIVEKIT_API_KEY=your_livekit_api_key LIVEKIT_API_SECRET=your_livekit_api_secret 📂 GitHub Repository 🔗 Source Code: [Add your GitHub repository link here] Make sure the public repository contains no API keys, phone numbers, caller information, database files, or other private data. 🔮 What I Want to Build Next There are several things I'd like to improve: Better multilingual support More natural conversation handling Improved interruption handling Better voice latency More learning-focused tools Advanced learner analytics Better specialist-agent routing Production-grade monitoring More robust phone-call infrastructure 🎯 Final Takeaway The biggest thing I learned during these 10 days is that building a voice agent is not just about connecting an LLM to a TTS model. A useful voice agent needs to: Listen → Understand → Reason → Act → Speak → Remember → Escalate → Improve Building Ava gave me hands-on experience with the complete voice AI pipeline and taught me a lot about real-time systems, APIs, databases, AI tools, agent architecture, debugging, and production challenges. I'm proud to have completed: 🚀 10 Days of Voice Agents - VoiceForBharat Edition Built with Murf Falcon + LiveKit + Gemini + Deepgram. A huge learning experience, and definitely just the beginning! 🎙️🤖 Top comments (0)