AI agent architecture: components of a system that survives real traffic

The core loop

Every AI agent, regardless of framework or implementation, executes a loop: receive input, decide what to do next, take an action, observe the result, and repeat until the task is complete or a stopping condition is reached. The complexity of a production agent is almost entirely in how that loop handles the cases where each step produces something unexpected. The happy path is easy. Production is the sum of all the unhappy paths.

The orchestrator

The orchestrator is the component that runs the loop. It holds the current state of the task, decides when to call the model, passes the assembled context, parses the model output, and routes the next action. Most agent frameworks - LangChain, LlamaIndex, AutoGen, Pydantic AI - provide an orchestrator. Building your own is appropriate for use cases where the framework abstractions produce more friction than value.

The orchestrator is where most production bugs live, because it is the component that has to handle every combination of model output and environment state. When a model returns a tool call with a malformed argument, when a tool returns an error partway through a multi-step task, when the task goal becomes unreachable due to an intermediate result, the orchestrator is what decides what happens next. That logic is almost never covered in a tutorial.

Tools and actions

Agents take actions by calling tools, functions with defined inputs and outputs that the model learns to invoke through examples in the system prompt. Tools might call an API, execute a database query, read a file, send an email, or invoke another model. Each tool is a potential failure point: it can time out, return an error, return unexpected data, or be called with incorrect arguments.

Production agent architecture defines what happens when any given tool fails: retry, escalate, stop, or proceed with partial information. Defining that behavior explicitly is what separates a prototype from a system. A prototype handles the success path. A production system handles every failure path with a deliberate decision about what to do next.

Tool design also affects model behavior. Tools with clear, specific names and well-typed schemas produce fewer incorrect invocations than tools with vague descriptions. The model's ability to choose the right tool is a function of how well the tool is specified, not just how capable the model is.

Memory and state

Agents need memory to complete multi-step tasks. Short-term memory is the conversation context within a single run. Long-term memory is information that persists across runs, stored in a database and retrieved when relevant. Episodic memory records what happened in past interactions and can be used to avoid repeating mistakes. Working memory is the structured state the agent builds up during a task - the intermediate results that inform subsequent actions.

Which types of memory a system needs depends on the task. Most agents need at least short-term and working memory. Systems that interact with the same user repeatedly benefit from long-term memory. The practical challenge is that memory grows without bound unless something manages it. Compaction strategies - whether truncation, summarization, or selective retrieval - all lose information in different ways. The decision of what to preserve and what to drop is part of the architecture, not an afterthought.

The planning problem

For complex tasks, agents need to plan before acting. Planning means decomposing a goal into steps, identifying dependencies, and ordering actions. Bad planning produces agents that take irreversible actions early, discover they needed information they did not collect, or pursue a sequence of steps that cannot reach the stated goal.

Good planning includes checkpoints where the agent validates intermediate results before proceeding. For tasks with significant consequences, human-in-the-loop checkpoints are part of the architecture, not a workaround. The question is not whether to include checkpoints but how to decide which steps warrant them. High-stakes actions, irreversible operations, and steps with high uncertainty are the obvious candidates. Building that decision logic into the orchestrator, rather than leaving it to the model, produces more reliable behavior.

Plan quality also degrades with task length. An agent that plans well for a three-step task will often produce a worse plan for a fifteen-step task, because longer plans have more surface area for compounding errors. One practical response is to plan one segment at a time, replanning at each checkpoint based on what actually happened rather than what was predicted.

Evaluation and monitoring

An agent that works in testing fails in production for reasons that were not in the test set. The solution is instrumentation: logging every model call, every tool invocation and its result, every decision point, and the final outcome. This log is what you need to diagnose failures, identify patterns in what goes wrong, and build regression tests from real cases. Without it, debugging a production agent failure is archaeology.

The most useful thing the log tells you is where the agent diverged from the expected path. If the model called the wrong tool, the log shows you the context it had at that moment and whether the tool definitions were ambiguous. If the agent looped, the log shows you what stopping condition it failed to trigger. (We covered the monitoring side in AI observability.)

Guardrails and stopping conditions

Agents that can take real-world actions need guardrails that limit what they can do without explicit approval. An agent that can send emails should not be able to send to arbitrary external addresses. An agent that can modify database records should have scope limits on what tables it can touch. An agent that can execute code should run in a sandbox. These are not optional for production systems. (We covered the broader topic in AI guardrails.)

Stopping conditions are equally important: define when the agent should stop and hand off to a human, not just when it should complete. An agent without a well-defined stopping condition will continue attempting to solve the task even when it has reached a state where further attempts are counterproductive. The stopping condition should encode the signals that indicate the task cannot be completed autonomously and a human needs to be involved.

What the architecture looks like in practice

A production agent at a company we have worked with processes customer onboarding documents. The orchestrator receives a document upload event and assembles the context: document content, customer record from CRM, and relevant policy rules retrieved via RAG. The model produces a structured extraction of key fields and a list of required follow-up actions. For each action, the orchestrator calls the appropriate tool, handles failures by logging and routing to a human review queue, and tracks completion state.

The agent runs autonomously for about 70 percent of cases. The other 30 percent go to human review with the agent output as a pre-populated draft. That 70 percent handled automatically represents the actual value of the system.

The architecture that produced those numbers was not the first version. The first version handled about 40 percent of cases and had no reliable failure routing. The path from 40 to 70 was mostly improving the orchestrator's failure handling, tightening tool schemas, and adding the human review queue as a first-class output. The model did not change. The architecture did.

Originally published at studiolabsai.com. Studio Labs builds production AI for enterprise teams. Book a call.