How to Build a Voice Agent with LangChain?

How to Build a Voice Agent with LangChain: Architecture, Streaming, Tools, and Production Patterns Building a voice agent is not simply a matter of connecting speech-to-text to an LLM and adding text-to-speech. A production voice agent has to solve a harder problem: How do you make an AI system listen, reason, use tools, remember context, and respond quickly enough that the conversation still feels natural? LangChain can handle the agent and tool-orchestration layer, but the realtime experience depends heavily on what happens around it. A practical architecture looks like this: User microphone ↓ Audio streaming ↓ Speech-to-Text (STT) ↓ Transcript / turn detection ↓ LangChain Agent ↓ Tools / APIs / Business Logic ↓ Streaming response ↓ Text-to-Speech (TTS) ↓ User hears response LangChain's current voice-agent documentation describes this as the "sandwich" architecture: STT → agent → TTS. The advantage is that each layer can be replaced independently, while the agent can continue using the broader LangChain ecosystem. What You Actually Need to Build Before writing code, separate the voice agent into five responsibilities: - Audio transport - moves microphone audio to the backend and audio responses back to the client. - Speech recognition - converts audio into text. - Agent reasoning - decides what the user wants and what action to take. - Tool execution - interacts with databases, CRMs, calendars, APIs, or internal systems. - Speech synthesis - converts the response back into audio. This separation matters because these components have different performance characteristics. For example, changing your TTS provider should not require rewriting your business logic. Similarly, changing the LLM should not require rebuilding your audio transport. That modularity is one of the strongest reasons to use a cascaded architecture instead of putting everything into one model. 1. Choose the Voice Architecture First There are two major ways to build a voice agent. Architecture A: STT → Agent → TTS Audio ↓ STT ↓ Text ↓ LangChain Agent ↓ Text ↓ TTS ↓ Audio This gives you control over every component. You can choose one STT provider, another LLM, and a completely different TTS provider. It also makes debugging easier because you can inspect the transcript, agent decision, tool call, and final response independently. The trade-off is additional infrastructure and potential latency. Architecture B: Speech-to-Speech Audio ↓ Multimodal Voice Model ↓ Audio This can reduce the number of moving pieces and can preserve more information about how something was spoken, such as tone. However, it can reduce your control over individual components and introduce provider-specific constraints. For business applications where tool execution, observability, provider flexibility, and deterministic workflows matter, the cascaded architecture remains highly practical. 2. Use Streaming Instead of Waiting for Complete Responses This is where many voice-agent implementations go wrong. A naive implementation waits for the entire chain: Record entire sentence ↓ Transcribe ↓ Wait for complete LLM response ↓ Generate complete audio ↓ Play response The user experiences one long delay. A streaming architecture instead looks like: Audio chunk ↓ STT starts immediately ↓ Transcript arrives ↓ Agent starts generating ↓ First response tokens arrive ↓ TTS starts ↓ Audio starts playing The system does not wait for every stage to finish before the next stage begins. LangChain's official voice-agent example uses asynchronous streaming and RunnableGenerator to connect STT, the agent, and TTS. The documentation notes that this pipeline can achieve sub-700 ms latency with suitable STT and TTS providers. The important lesson is: Realtime voice is primarily a pipeline-design problem, not just a model-selection problem. Research on realtime voice agents similarly identifies streaming and pipelining across STT, LLM, and TTS as a central mechanism for reducing perceived latency. 3. Create the LangChain Agent Once speech has been converted into text, the voice layer can hand the request to a normal LangChain agent. Current LangChain applications use create_agent as the primary entry point. A minimal agent can look like this: from langchain.agents import create_agent def check_order_status(order_id: str) -> str: """Return the current status of an order.""" return f"Order {order_id} is currently being processed." agent = create_agent( model="openai:gpt-5.4", tools=[check_order_status], system_prompt=""" You are a customer support voice agent. Keep spoken responses short. Ask for missing information instead of guessing. Use tools whenever the user asks for account-specific information. """ ) The important part is not the five lines of code. It is the tool boundary. A voice agent should not directly manipulate your database or business systems through arbitrary model-generated text. Instead: User: "Where is order 4821?" ↓ Agent ↓ check_order_status("4821") ↓ Business system ↓ Structured result ↓ Agent ↓ "Your order is currently being processed." LangChain agents can reason over available tools and execute them as part of the agent loop. The current agent implementation is built on LangGraph's runtime. 4. Design Tools for Voice, Not Just for Chat This is an overlooked part of voice-agent engineering. A tool that works well for a text chatbot may be poorly designed for a voice agent. For example, avoid giving the agent a tool that returns: {"customer_id": 1827, "subscription_status": "active", "plan": "enterprise", "billing_cycle": "annual", "last_payment": "...", "payment_method": "..."} if the only thing the user asked was: "Is my subscription active?" Instead, make the tool return information that the agent can quickly reason over. def get_subscription_status(customer_id: str) -> str: """Check whether a customer's subscription is active.""" ... The voice agent can then respond: "Yes, your subscription is active." The rule is simple: Design tools around decisions, not database tables. This reduces unnecessary reasoning and makes spoken responses easier to control. 5. Keep Spoken Responses Short A language model optimized for written chat can produce paragraphs. A voice agent should not. Compare: Chatbot response: "Certainly. I can help you with that. According to the information available in your account, your order has been processed successfully and is currently in transit. You can expect delivery within the next two to three business days..." Voice response: "Your order is in transit. It should arrive within two to three business days." Voice requires a different response policy. A useful system instruction is: You are a voice assistant. Speak naturally and concisely. Prefer one or two sentences per response. Do not read JSON, URLs, IDs, tables, or long lists aloud. Ask one question at a time. If a tool fails, explain the problem briefly and offer the next action. Never invent information that is unavailable from a tool. This is not merely prompt optimization. It is interface design. 6. Add Conversation Memory Carefully Voice conversations become awkward if the agent forgets what was said five seconds earlier. Consider: User: "I want to book an appointment tomorrow." Agent: "What time?" User: "Around 4." Please ensure the agent understands that "4" refers to the appointment. LangChain's voice-agent example uses conversation state with a checkpointer and a unique thread ID so the agent can retain context across turns. Conceptually: User ↓ Voice session ID ↓ Conversation state ↓ LangChain agent ↓ Response For a production system, distinguish between: Short-term conversation state Things said during the current call. Examples: - user's name - requested appointment time - current order number - selected product Long-term business memory Information that should survive the call. Examples: - customer preferences - previous interactions - account information Do not put every piece of customer data into the LLM's conversation history. Retrieve what is needed for the current decision. 7. Handle Interruptions This is one of the biggest differences between a chatbot and a voice agent. Imagine the agent is saying: "Your appointment is scheduled for Thursday at-" The user interrupts: "Actually, make that Friday." A real voice interface should stop speaking. That means your system needs to support barge-in. A simplified flow is: Agent speaking ↓ User starts talking ↓ Detect interruption ↓ Stop TTS playback ↓ Cancel/ignore remaining audio ↓ Process new user input Without interruption handling, the system feels less like a conversation and more like an IVR reading a script. This is why audio transport, turn detection, and cancellation logic are just as important as the LLM. 8. Use WebSockets for Browser-Based Streaming For a browser-based implementation, WebSockets are a practical transport layer. The client captures microphone audio: Browser microphone ↓ PCM audio chunks ↓ WebSocket ↓ Backend The backend sends synthesized audio back through the same connection: Backend ↓ TTS audio chunks ↓ WebSocket ↓ Browser ↓ Speaker LangChain's reference voice application uses WebSockets for bidirectional audio streaming and notes that the same general architecture can be adapted to telephony or WebRTC. The important design decision is to keep the transport layer independent from the agent. Your agent should not care whether the request came from: - a browser - a mobile application - a phone call - a WebRTC client It should receive an input event and return agent events. 9. Connect the Pieces with an Async Pipeline A simplified LangChain pipeline can conceptually look like: from langchain_core.runnables import RunnableGenerator pipeline = ( RunnableGenerator(stt_stream) | RunnableGenerator(agent_stream) | RunnableGenerator(tts_stream) ) Each stage consumes and produces a stream. STT events ↓ Agent events ↓ TTS events This is more useful than treating the voice agent as one giant function. Each stage can