Graph Engineering Explained: The Missing Fifth Layer of AI Agent Architecture
Every "my agent isn't working" postmortem starts the same way: someone rewrites the prompt. Adds a constraint. Adds an example. Ships it again. Three iterations later the agent still can't hold up in production, and the team is quietly out of ideas - because the prompt was never the layer that broke. There are five control layers standing between a raw model call and a system you can actually trust with a business outcome: prompt, context, harness, loop, and graph. Most teams staff and instrument only the first one or two. The failures that show up in production - wrong tool called, same mistake retried forever, output routed to the wrong reviewer - live almost entirely in the layers nobody named. Graph engineering is the newest and least understood of the five: it's the layer that decides which component runs next, when agents work in parallel versus in sequence, and where a human has to sign off before anything expensive or irreversible happens. This piece breaks down all five layers, works through a single production failure end to end, and shows where evals fit as the measurement system running through every one of them. The mental model: five rings around the model MODEL CALL = prompt + context AGENT = model call + harness + loop SYSTEM = agents + deterministic steps + humans, connected by a graph EVALS = evidence that every layer actually works Prompt and context sit closest to the model. Harness and loop turn a model call into something that can act and recover. Graph turns a collection of agents, functions, and human checkpoints into a coordinated system. None of these layers replace each other - they're concentric controls, not pipeline stages, and a production agent uses all five simultaneously. The weakest layer sets the ceiling on how reliable the whole thing is, no matter how good the other four are. | Layer | Controls | Fails as | |---|---|---| | Prompt | Role, goal, constraints, output contract | Ambiguous instructions | | Context | What reaches the window: docs, history, tool results | Missing or noisy evidence | | Harness | Tools, file/shell access, sandboxing, permissions | Overprivileged or unsafe actions | | Loop | Retry policy, validators, stop conditions, escalation | Infinite retries on the same mistake | | Graph | Routing, parallelism, recovery paths, human gates | Work reaching the wrong next step | A production failure, diagnosed layer by layer Consider a coding agent built to fix low-risk defects in an internal payments service. The prompt is reasonable: inspect the issue, avoid unrelated changes, run the tests, return a PR summary. On a clean sample repo, it works. On the real repository, it falls apart in four distinct ways: - It misses an architecture decision buried in the docs - a context failure. - It runs a shell command with a broader scope than intended - a harness failure. - It retries the same failing test without changing its hypothesis - a loop failure. - It sends the pull request down the wrong review path - a graph failure. The natural instinct is to ask "how do we improve the prompt?" That's the wrong question. Only one of these four failures traces back to the instruction layer - and it isn't the one that caused the damage. Each failure needs a fix in the layer that actually owns it, not a paragraph bolted onto the system prompt. Layer 1 - Prompt: steers one model call Analyze the reported defect and propose the smallest safe fix. Do not change unrelated behavior. Return the root cause, files changed, test evidence, and residual risk. Stop and ask for approval if the fix changes an external contract. The unit being optimized here is a single model interaction. A stronger prompt reduces ambiguity, but it cannot supply a missing design document, restrict a dangerous tool, or decide who reviews the output. In an agent system, the prompt is the steering wheel - not the car. Layer 2 - Context: what the model can actually see Ask a model to summarize risk in an 80-page contract. Dumping the whole document into the window and retrieving the liability, indemnification, termination, and data-use clauses (plus the org's risk policy) produce two very different answers from the same prompt. The instruction didn't change - the evidence available to answer it did. For the coding agent, the missing architecture decision is a retrieval problem. Rewording the prompt might paper over one test case; fixing context assembly fixes the whole class of failure. Layer 3 - Harness: the runtime envelope The harness is everything around the model call: tools, file access, shell access, MCP connections, sandboxing, permissions, timeouts, logging, approval boundaries. The model can decide "I need to run the tests" - the harness decides whether that's even possible, which commands are allowlisted, which directory is visible, and what gets recorded. MCP standardizes how an agent connects to tools; it does not decide that an agent deserves production write access. Identity, least privilege, and approval policy still belong to the host and its surrounding control plane. This is usually the first layer a security team asks about, and it's exactly where the broad shell command should have been caught. Layer 4 - Loop: the retry contract Loop engineering owns the cycle - act, observe, evaluate, adjust, repeat - plus retry policy, validators, completion criteria, budgets, and escalation rules. It's a distinct concern from the harness: - Harness asks: Can the agent execute the test, in which sandbox, with what timeout? - Loop asks: Does a failed test trigger another attempt, what has to change before retrying, how many attempts are allowed, and what counts as done? You can have a perfectly sandboxed, fully logged harness and still watch an agent burn its entire budget retrying the identical failed fix. The coding agent's repeated test failure needed a new-hypothesis requirement and a retry cap - not broader filesystem access. Layer 5 - Graph: coordinating the system Graph Engineering is the operational paradigm for building complex AI agents and multi-agent systems by representing their workflows as explicit stateful graphs rather than relying on unstructured, single-agent loops or linear prompt chains. Instead of letting an LLM autonomously decide every execution step in an unpredictable loop ("prompt and pray"), graph engineering imposes architectural boundaries. It treats the overall task as a state machine where nodes execute discrete logic (LLM calls, tool execution, validation), edges direct routing decisions, and a schema-defined state persists throughout the lifecycle. Graph engineering controls the topology of the whole workflow. Nodes can be agents, deterministic functions, evaluators, or human gates; edges define sequencing, routing, parallel branches, recovery paths, and where the loops from layer 4 actually live. Loop asks "how does this one agent keep working?" Graph asks "which component runs next, and how does the system coordinate?" flowchart LR A[Triage] --> B[Planner] B --> C[Coding Agent] C --> D[Deterministic Tests] D -->|pass| E[Security Reviewer] D -->|fail| C E --> F{Human Approval} F -->|approved| G[Merge] F -->|rejected| B That's the fix for the coding agent's fourth failure: an explicit route from code change to tests, to security review, to human approval before merge - instead of an implicit hope that the right person eventually sees it. LangGraph frames itself as a low-level orchestration runtime for exactly this: mixing deterministic steps with model-driven steps while preserving state, durable execution, and human interrupts. The useful idea isn't "draw boxes and arrows" - it's splitting responsibilities that a single overloaded chat session was quietly doing all at once (plan, research, write, and approve its own work), and keeping a human where mistakes get expensive. Graph complexity isn't free, and the data backs that up: Anthropic reported its multi-agent research system beat a single-agent setup by 90.2% on an internal breadth-first research evaluation - but the multi-agent runs consumed roughly 15x the tokens of a normal chat interaction. That number is specific to Anthropic's research workload, not a universal multiplier, but it captures the trade-off precisely: graphs earn their complexity only when the task's value and parallelism justify the bill. Reach for a graph because the workflow genuinely branches, not because orchestration frameworks are the interesting part of the stack right now. Core Pillars of Graph Engineering [ Shared Typed State Object (e.g., Pydantic / TypedDict) ] | +----------------------+----------------------+ | | v v [ Node: LLM / Tool / Task ] ---------> [ Conditional Edge ] | | +----------------------+----------------------+ | v [ Node: Validator / Human Checkpoint ] 1. State Management The explicit data structure passed through every execution step in the graph. - Typed Schemas: Defines exact variables, tool outputs, message histories, and system metadata. - Reducers: Functions that determine how state field updates from parallel or sequential steps are merged (e.g., appending items to a list vs. overwriting a variable). 2. Nodes (Units of Execution) Self-contained, bounded steps inside the system. A node takes the current state, performs logic, and returns a state patch. - Agentic Nodes: Specialized LLM calls tailored to a single role (e.g., Researcher, Refiner, Evaluator). - Deterministic Nodes: Standard code executions (API calls, data parsers, formatting utilities). - Validation Nodes: Output parsers and schema checkers that evaluate prior steps. 3. Edges (Control Flow & Routing) Rules that connect nodes and govern system transitions. - Fixed Edges: Direct, deterministic routing from Node A to Node B. - Conditional Edges: Dynamic routing based on LLM outputs or state evaluation (e.g., if confidence Evaluate -> Revise -> Evaluate). The concern that cuts across all five: evals There's a sixth thread running through every layer, and it
Comments
No comments yet. Start the discussion.