Voice Agents in Microsoft Foundry: Inside the Realtime Speech-to-Speech Architecture

Table of Contents

• What Problem Voice Agents Actually Solve
• Where Voice Agents Sit in the Foundry Agent Taxonomy
• Architecture: From WebSocket to Model and Back
• Defining a Voice Agent
• Turn Detection, Barge-In, and Why Silence Duration Matters
• Function Calling Over a Realtime Session: The Deferred-Response Pattern
• MCP Tools, Toolbox Tools, and System Tools in Voice Context
• Bring-Your-Own-Model (BYOM): Managed vs Self-Deployed
• Persistence: Conversations, Transcripts, and Audio Playback
• A Real-World Scenario: A Voice-Driven Support Triage Agent
• Production Considerations
• Security Considerations
• Performance, Scale, and Latency Budgets
• Cost Considerations
• Common Mistakes and Pitfalls
• Alternatives and Trade-offs
• Practical Recommendations
• Conclusion
• References

Why this matters

Every chat-based agent you've built so far has had the luxury of a request/response boundary. A user sends a message, your agent thinks for however long it needs, calls a tool, thinks some more, and returns an answer. Nobody is standing there in real time waiting for the next word.

Voice breaks that contract completely. A caller doesn't pause while your agent decides whether to invoke a get_weather function. They keep talking, they interrupt, they say "actually never mind" halfway through a sentence, and they expect a natural reply within a few hundred milliseconds - not because your product spec says so, but because that's how human conversation works neurologically. Silence past ~300ms reads as "did it hang up?"

Microsoft Foundry's answer to this problem is Voice Agents (currently in preview), a first-class agent kind sitting alongside prompt agents, hosted agents, workflows, and external agents in the same project_client.agents management surface. But the interesting engineering isn't that Foundry added a voice mode - it's how it had to restructure agent execution to make tool calling, turn detection, and interruption handling work over a persistent WebSocket instead of a stateless HTTP call.

This article is a deep, implementation-level look at that architecture: what happens on the wire, why function calling requires a deferred-response pattern you won't find in text agents, how turn detection and barge-in actually work, and what production considerations (security, cost, scale, failure modes) look like once you put a live microphone in front of an LLM.

If you've been building text and hosted agents in Foundry (Responses/Invocations protocols, MCP toolboxes, the Agent Optimizer), this is the piece that completes the picture: voice is not "chat with an audio codec bolted on." It's a genuinely different runtime model.

What Problem Voice Agents Actually Solve

Before Foundry Voice Agents, if you wanted a speech-to-speech assistant you had two realistic paths:

• Cascaded pipeline - Speech-to-text (Azure Speech / Whisper) → LLM completion → text-to-speech. You own every hop, every buffer, every latency budget, and every failure mode independently.
• Raw Realtime API - Talk directly to a realtime model's WebSocket endpoint (e.g., gpt-realtime) yourself, hand-rolling session state, reconnection, tool dispatch, and persistence.

Both work, but both push a huge amount of "voice agent plumbing" onto every team that wants to ship one: VAD tuning, barge-in handling, transcript persistence, tool-call race conditions, and governance (who can call what tool, from which agent). Multiply that by every team in an enterprise building a different voice assistant and you get a lot of reinvented, subtly-buggy wheels.

Foundry Voice Agents fold that plumbing into the platform. The agent is a versioned, governed resource - the same object model you already use for prompt and hosted agents - but its definition carries voice-specific concerns (audio codecs, turn detection thresholds, output voice) and its runtime is a managed realtime orchestrator instead of a single request handler. You still write the tool logic and the business rules; the platform owns the wire protocol, the turn-taking, and (optionally) the transcript/audio persistence.

Where Voice Agents Sit in the Foundry Agent Taxonomy

Foundry's project_client.agents surface is unified across kinds:

from azure.ai.projects.models import AgentKind
for item in project_client.agents.list(kind=AgentKind.VOICE):
    print(item.name)

The same create_version / get_version / disable / enable / delete_version lifecycle you use for prompt agents (create_from_prompt) or hosted agents applies to voice agents with kind="voice". This matters architecturally: it means voice agents inherit whatever governance model Foundry projects already enforce - RBAC on the project, agent versioning and rollback, and the same audit trail - rather than living as a bolted-on, parallel resource type with its own permission model.

What's different is the runtime surface exposed for actually talking to one:

• project_client.agents - management (create, version, list, enable/disable, delete). Identical shape to other agent kinds.
• project_client.beta.voice_agents.realtime - the live WebSocket connection for holding a conversation.
• project_client.beta.voice_agents.conversations - a read-only API for pulling back persisted transcripts and audio after the fact.

Note the beta namespace and the requirement to construct the client with allow_preview=True. This is a genuine preview feature - expect API shape changes before GA, and don't build irreversible production dependencies on field names yet.

Architecture: From WebSocket to Model and Back

Here's the request flow for a live voice turn, spelled out because it explains almost every design decision downstream:

Two things stand out compared to a text agent:

• First, the connection is a session, not a call. You connect(agent_name=...) once and hold it open for the duration of the conversation. Everything - user turns, model responses, tool calls, turn detection events - flows as typed events over that single socket (conn.recv()), not as discrete HTTP requests.
• Second, tool execution is split into two categories with fundamentally different trust models:
  • Client-executed tools (function type) - the service pauses generation, sends you the call, and waits for your application process to send the result back over the same socket. Your code, your infrastructure, your latency.
  • Service-executed tools (system, mcp, toolbox) - the platform calls out to a remote MCP server or an internal Foundry Toolbox on your behalf, without a round trip through your client process.

That split is not cosmetic. It's the difference between "the caller's phone app can hang or crash mid-tool-call" and "the tool call happens entirely within Foundry's infrastructure regardless of client health." Design your tool architecture around which category each capability belongs in.

Defining a Voice Agent

A minimal voice agent definition looks like this:

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
    VoiceAgentDefinition,
    VoiceAgentAudioConfig,
    VoiceAgentAudioOutputConfig,
    VoiceModelType,
    VoiceOutputModality,
    VoiceType,
)

endpoint = "https://<your-project>.services.ai.azure.com/api/projects/<project-name>"

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(
        endpoint=endpoint,
        credential=credential,
        allow_preview=True,
    ) as project_client,
):
    definition = VoiceAgentDefinition(
        model_type=VoiceModelType.MANAGED,  # "managed" = service-hosted realtime model
        model="gpt-realtime",
        instructions="You are a friendly voice assistant. Keep replies short and natural.",
        audio=VoiceAgentAudioConfig(
            output=VoiceAgentAudioOutputConfig(
                voice="en-US-AvaNeural",
                voice_type=VoiceType.AZURE_STANDARD,
            ),
        ),
        output_modalities=[VoiceOutputModality.AUDIO],
        # store=True persists the transcript + audio for later retrieval.
        # Defaults to False - nothing is retained unless you opt in.
        store=True,
    )
    created = project_client.agents.create_version(
        agent_name="MyVoiceAgent",
        definition=definition,
    )
    print(f" Created version: { created . version } ")

A few details worth internalizing:

• output_modalities controls whether the agent replies with synthesized audio (AUDIO) or plain text transcripts (TEXT). Text-only output is genuinely useful for automated testing of a voice agent's reasoning without paying for or waiting on speech synthesis - see the function-tool sample later, which deliberately uses TEXT output for exactly this reason.
• Versioning is immutable. Every create_version call - even one that only changes the system instructions - produces a new, independently addressable version. There is no in-place mutation of a live agent version. This is the same model prompt agents use, and it means you can roll back a voice agent's personality/tool config as cleanly as you'd roll back a container image tag.
• store defaults to False. Nothing is retained unless you explicitly opt in - an intentional privacy-by-default choice given that voice sessions inherently capture biometric-adjacent data (a person's actual voice).

Turn Detection, Barge-In, and Why Silence Duration Matters

The richer configuration surface lives in VoiceAgentAudioInputConfig:

from azure.ai.projects.models import (
    RealtimeAudioFormatsAudioPcm,
    VoiceAgentAudioInputConfig,
    VoiceAgentInputTranscription,
    VoiceAgentInputTranscriptionModel,
    VoiceAgentServerVadTurnDetection,
)

audio_input = VoiceAgentAudioInputConfig(
    format=RealtimeAudioFormatsAudioPcm(rate=24000),
    turn_detection=VoiceAgentServerVadTurnDetection(
        threshold=0.5,  # sensitivity of "is this speech" classification
        prefix_padding_ms=300,  # audio captured just *before* speech is detected,
        # so the first phoneme of a word isn't clipped
        silence_duration_ms=500,  # how long the caller must be silent before
        # the service treats the turn as "done" and triggers a response
    ),
    transcription=VoiceAgentInputTranscription(
        model=VoiceAgentInputTranscriptionModel.WHISPER1
    ),
)

This is server-side VAD (voice activity detection) - the orchestrator, not your client, decides when the caller has finished a turn. That's a deliberate architectural choice: turn-taking is genuinely hard to get right (accents, background noise, thinking pauses vs. "I'm done talking" pauses), and centralizing it in the platform means every voice agent in your organization gets the same tuned behavior instead of every team hand-rolling energy-threshold VAD in JavaScript.

The two knobs that matter most in practice:

• silence_duration_ms is your latency/false-interruption trade-off. Too low (e.g., 200ms) and the agent jumps in during a caller's natural mid-sentence pause. Too high (e.g., 1200ms) and every reply feels sluggish. 500ms is a reasonable starting point for conversational English; expect to tune it per locale and per use case (a support triage bot tolerates more pause time than a rapid-fire trivia game).
• prefix_padding_ms protects against clipped transcription. Speech classifiers need a few frames to become confident that speech has started, and without padding you lose the consonant or syllable that triggered the detection.

Barge-in - the caller interrupting the agent mid-sentence - is a first-class behavior in the bidirectional audio sample (voice_agent_realtime_audio_conversation_async.py), not something you implement yourself. When server VAD detects new speech while the agent is still speaking, the orchestrator truncates the in-flight response and starts listening. If you've ever built this by hand with raw WebRTC and an LLM, you know how much edge-case handling that one sentence is quietly doing (audio buffer truncation, response cancellation, avoiding echo-triggered false interruptions from the agent's own voice bleeding into the mic).

Function Calling Over a Realtime Session: The Deferred-Response Pattern

This is the part of voice agents that will bite you if you port over your intuition from text-based tool calling, so it's worth walking through carefully.

In a text agent (Responses or Invocations protocol), tool calling is naturally sequential: the model emits a tool call, execution pauses, you run the tool, you send the result back, generation resumes. There's no ambiguity about ordering because everything is a single logical turn.

In a realtime voice session, the model is continuously capable of receiving events, and a response.create() call while a function-call response is still finishing produces a concurrent-response error - the service rejects overlapping generation requests on the same conversation.

The correct pattern, straight from Foundry's own sample code, is:

def _run_turn_with_tool_support(client, agent_name, prompt):
    with client.beta.voice_agents.realtime.connect(agent_name=agent_name) as conn:
        conn.conversation.item.create(
            item=RealtimeConversationItemMessageUser(
                type=RealtimeConversationItemType.MESSAGE,
                content=[
                    RealtimeConversationItemMessageUserContent(
                        type="input_text",
                        text=prompt,
                    ),
                ],
            ),
        )
        conn.response.create()
        # Tool outputs are collected but NOT sent immediately -- sending them
        # while the function-call response is still in flight races with the
        # service and can produce a concurrent-response error.
        pending_tool_outputs = []
        while True:
            event = conn.recv(timeout=45)
            if isinstance(event, RealtimeServerEventResponseFunctionCallArgumentsDone):
                args = json.loads(event.arguments)
                result = (
                    get_weather(**args)
                    if event.name == "get_weather"
                    else json.dumps({"error": f"Unknown tool: {event.name}"})
                )
                pending_tool_outputs.append((event.call_id, result))
            elif isinstance(event, RealtimeServerEventResponseDone):
                # Only NOW, after this response has fully completed, is it
                # safe to submit tool outputs and request the next respo

The article body ends here, mid-sentence, in the original source. No further content was provided to complete the deferred-response pattern explanation.