Designing Smart AI Agents: Architecture Patterns That Survive Production

A practical field guide to agent topologies, state design, and the failure modes that separate a demo from a system that survives 90 days in production. Six months ago, a logistics company in Dubai flew me in to look at their "autonomous customer operations" pilot. The vendor demo was stunning. An agent quoted delivery timelines, resolved address discrepancies, and flagged high-risk shipments for human review. In the controlled demo, it resolved 94% of test cases without a human in the loop. I asked for the production numbers. A pause. Then the CTO pulled up a dashboard. On real traffic - about 4,000 tickets a day, messy addresses, late tracking feeds, angry customers - the same agent resolved 11% of cases, and it hallucinated a delivery promise onto at least a dozen of the rest. Some of those promises cost the company real money in refunds and re-shipping. The demo was not fake. The model was fine. The problem was that nobody had designed the system's architecture. They had pointed a capable model at a prompt, wrapped it in a loop, and called it an agent. This article is the pattern language I wish that vendor had used: the topologies, the state design, the tool contracts, and the failure modes that decide whether an agent survives production. By the end you will be able to look at any agent project, name the pattern it is using, and - more importantly - say whether it is the right one. First, Kill the Word "Agent" - Talk About Topology "Agent" is a marketing word. "Topology" is an engineering word. A topology is the shape of your system: how many reasoning loops exist, how they talk to each other, who owns the state, and who decides what happens next. When a production agent collapses, it is almost never the model's fault. It is a topology that did not match the task. Every agent architecture, no matter how clever the slideware, is one of six topologies. Learn to spot them, because each has a cost profile, a failure mode, and a narrow range of tasks it is genuinely good at. 1. The Single Loop One model, one context window, a set of tools, a while loop. This is the default and the workhorse. The system prompt holds the goal, the context window holds working state, tools give it hands, and budget counters stop it from running forever. Cost: lowest. Latency: lowest. Good for: narrow, well-scoped tasks - balance lookups, form extraction, single-domain Q&A with tools. 2. The Router A small, fast model classifies the request and dispatches it to one of several specialized handlers. A ticket-triaging router sends payment disputes to a refund workflow, delivery questions to a tracking tool, and everything else to a general agent. Cost: low. Latency: adds one cheap call. Good for: high-volume traffic where most requests are one of a few known shapes. This is the most underrated pattern in production, and the one I reach for first. 3. Orchestrator-Worker One orchestrator decomposes a task into subtasks and hands each to a worker agent (or a plain function, or a search job). Workers return results; the orchestrator synthesizes. This is what people actually mean when they say "multi-agent," and most of the time it is one orchestrator with several specialized workers. Cost: medium. Latency: medium. Good for: report generation, research, code review - tasks with a natural breakdown. 4. Hierarchical Agents manage agents. A lead orchestrator spawns sub-orchestrators, each managing its own workers. This is how you scale orchestrator-worker to genuinely huge tasks, and it is also where complexity and cost start to compound. Cost: high. Latency: high. Good for: enterprise research pipelines with thousands of documents. Usually a mistake for anything pattern 3 handles. 5. Peer Team Multiple agents with equal standing converse or work in parallel toward a shared goal - the classic CrewAI and AutoGen picture: a researcher, a writer, and a critic arguing over a document until they agree. Cost: high - every peer turn is a full model call and coordination overhead is real. Latency: high. Good for: creative drafting and debate-style tasks. Bad for: anything with a deadline and a strict budget. 6. The State Machine (Workflow) No loop at all. A directed graph of steps - query, validate, charge, confirm - where each step is deterministic or model-assisted. LangGraph's graph model and n8n's node model are this pattern wearing graph paper. Cost: lowest per step. Latency: predictable. Good for: anything that is 80% a known process with a few fuzzy decision points. Most "agent" use cases are secretly this, and it is the most honest pattern in the list. Here is the quick reference I put in front of clients: | Pattern | Loop? | Cost | Failure mode | Best for | |---|---|---|---|---| | Single loop | yes | low | context creep | narrow, scoped tasks | | Router | no | low | bad classifier | high-volume triage | | Orchestrator-worker | yes | medium | handoff context loss | research, reports | | Hierarchical | yes | high | exponential cost | huge decompositions | | Peer team | yes | high | coordination chatter | drafting, debate | | State machine | no | low | rigid on exceptions | known processes | The most useful question I ask before writing any code: is this task a process with a few judgment calls, or an open-ended goal with unknown steps? Process → state machine. Open-ended → single loop or orchestrator-worker. Almost never a peer team on day one. State: The Part Everyone Forgets Now the part that kills more production agents than any topology choice: state. An agent's state is everything it carries between steps - the task definition, what it has already tried, what it has ruled out, the results of tool calls, and the budget it has left. If state lives only in the model's context window, you have a memory problem: context windows are bounded, noisy, and easy to poison. If state lives in your database, you have an engineering problem: every step needs a save, a load, and a version. Here is the rule I now enforce with clients. Working state (what is on the model's desk right now) goes in the context window, trimmed ruthlessly. Durable state (what this task has accomplished, across retries and restarts) goes in a store - Postgres for structured task state, a vector store for retrieved knowledge, Redis for ephemeral job state. Every step is a pure function of durable state plus the model's decision. That one discipline - "state in the store, not in the prompt" - fixed more agent projects than any model upgrade I have ever shipped. Concretely, a task row in Postgres looks like this: CREATE TABLE agent_tasks ( id uuid PRIMARY KEY, pattern text NOT NULL, -- which topology goal text NOT NULL, status text NOT NULL DEFAULT 'queued', step_count int NOT NULL DEFAULT 0, tool_calls jsonb NOT NULL DEFAULT '[]', result jsonb, created_at timestamptz NOT NULL DEFAULT now() ); Every tool call is appended to tool_calls . If the process crashes, a worker picks up the row and replays from step_count . That is the entire secret of "reliable" agents: they are just jobs that can resume. Tool Design: Descriptions Are Contracts I keep saying tools are the agent's hands, but the part people get wrong is the description. The model reads your tool description and decides whether to use the tool. Write a lazy description and the model will misuse it in production, every single time. Treat the description as a contract with three clauses: - What it does. "Fetches the current available balance for a verified account." - When to use it. "Call this when the customer asks about money they have or owe. Do not call it for transaction history - that is get_transactions ." - What it returns. "Returns {balance: number} . Returns an error object if the account is not verified." One more rule: validate inputs server-side before execution. The model's arguments are model output - they can be wrong, and in adversarial inputs they can be malicious. A SQL injection string smuggled through a tool argument is not a joke; it is a Tuesday. A Working Orchestrator-Worker, Minimal Here is the smallest orchestrator-worker I would ship, with the state discipline above. No framework - just Postgres, a queue, and two model calls per task. import json from typing import Any def orchestrator(task: dict) -> str: plan = llm_call( "You are a research lead. Split this task into 3-5 subtasks " "that can be executed independently. Return JSON.", task["goal"], ) subtasks = json.loads(plan)["subtasks"] results = [] for sub in subtasks: results.append(worker(sub)) # worker may call tools save_task_state(task["id"], results) # durable state every step return llm_call( "You are a synthesis editor. Combine these subtask results " "into one coherent answer for the original task.", json.dumps({"goal": task["goal"], "results": results}), ) def worker(subtask: dict) -> Any: # deterministic routing: one tool call, one model pass return run_tool(subtask["tool"], subtask["args"]) def save_task_state(task_id: str, results: list) -> None: # UPDATE agent_tasks SET tool_calls = $1 WHERE id = $2 pass Run this against real traffic and you will find the handoffs - the exact spots where context gets lost and tasks stall. That is the point: you want your failures in the handoff layer, because handoffs are cheap to instrument and cheap to fix. A hallucinated subtask decomposition, by contrast, is expensive to catch and expensive to repair. Production Reality: The Failure Modes That Actually Hurt After a year of shipping these systems across fintech, logistics, and support clients, here is my honest list of what breaks, ranked by how much it hurts: - Silent overreach. The agent does something you never authorized, confidently. It sends the email, applies the discount, closes the ticket. Fix: a permission layer - read-only tools are free; mutating tools require approval or a hard policy. - Context creep. Every step appends to the prompt, so by step 9 the model is reading a wall of its own noise. Fix: trim aggressively, summarize ol