How to Build AI Evals for Tool-Calling Agents

Every other week it feels like a new model shows up with a shiny score on some "trust me bro" benchmark. The numbers climb, people call it smarter, and suddenly you're ready to switch. If you build applications on top of these models, you might catch yourself assuming that higher benchmark score means better agent performance. But benchmarks measure narrow skills under controlled conditions; your agent has a specific job to do. If you're building a customer support agent, you don't care about a few extra points on a reasoning test. You care about whether it calls the right tool, passes the right arguments, and avoids risky or redundant actions. That's the behavior that actually matters. Testing a tool-calling agent means testing its decisions, not just its words. An agent can sound perfectly convincing while quietly messing things up underneath. When it answers "Your refund has been processed", the response sounds fine but did it check the right order? Refund the correct amount? Accidentally hit the refund API twice? The final response won't tell you. What you actually need is an eval suite: a set of automated tests that score how well an agent performs. Eval is short for evaluation: each test runs the agent on a scenario, asks a grader "was this behavior correct?", and turns the answer into a score you can track and gate on. Traditional software tests and prompt-and-response evals both fall short here. A unit test asserts that a function returns the right value for a given input. An agent, though, makes a sequence of decisions (which tool to call, with what arguments, in what order), and each decision is non-deterministic. The same prompt can produce a different tool-calling path on every run. A single manual test tells you what the agent can do, not what it typically does. So the eval suite should answer three questions: - Selection: Did the agent pick the right tool for the request? - Trajectory: Did it call the right tools, with the right arguments, in the right order, without wasted or dangerous calls? - Outcome: Is the final answer correct and grounded in what the tools returned? In this guide, you'll build exactly that. Using Mastra, you'll create a customer support agent that can look up orders, process refunds, and escalate to a human. Then you'll write a layered eval suite for it: - Quick Checks: zero-LLM, deterministic assertions like "the agent must call lookup_order " and "no tool call may error" - Gates and verdicts: hard pass/fail requirements that fail a run outright - Trajectory scorers: validate the full sequence of tool calls, including arguments, step budgets, and blacklisted tools - LLM-as-a-judge scorers: semantic grading for the cases where exact matching is too rigid - A Vitest suite: so the whole thing runs in CI on every change Before the code: what an eval is Before we start typing, let's get one mental model straight, because it's the vocabulary the rest of the guide (and Mastra's API) assumes. An eval is three things put together: a test case, an expected behavior, and a grader. Input: "I want a refund for order 1001" Expected behavior: Look up order 1001, then process the refund. Grader: Did those things happen, in that order, with the correct order ID? Score: 1 if correct,0 if not. Unlike a traditional unit test, you usually run an agent through the same input multiple times, because an agent's behavior isn't deterministic. The same prompt can call different tools on different runs, so you score many runs and average the results rather than trusting a single pass. A handful of terms come up constantly, and it helps to pin them down now: | Concept | Meaning | |---|---| | Test case / data item | The scenario you give the agent (input ) | | Expected behavior | What you believe the agent should do | | Scorer / grader | The mechanism that judges whether the agent did it | | Eval suite | The collection of test cases plus graders | | Gate | A requirement that must pass, or the run fails | | Threshold | A minimum acceptable average score | Two of these matter more than the rest, so let's call them out. A gate is a hard requirement: if it doesn't pass, the whole run is failed , full stop. A threshold is a softer quality bar, like "average relevancy above 0.8", that a run can miss and still be usable. Keep that difference in mind; we devote a whole section to it below. One last principle, and it's arguably the most important lesson in the guide: a grader is only meaningful relative to the scenario being graded. The scorer "the agent must call lookup_order " is exactly right for an order-status request, but it should fail for the question "What's the capital of France?" and that failure isn't an agent bug. It just means that scorer belongs to a different scenario than the off-topic one. The way forward is never a weaker, vaguer checker; it's organizing your test cases so each scenario gets graders that match its own expectations. Why testing tool-calling agents is different When a user sends a request to an agent, several decisions happen in sequence. The agent decides which tool (if any) to call, builds the arguments, executes the call, reads the result, and either calls another tool or synthesizes a final answer. Each step introduces a distinct failure mode: - Wrong tool selection. The user asks for a refund, and the agent calls lookup_order but neverprocess_refund , then claims the refund was processed anyway. - Wrong arguments. The agent calls the right tool but passes orderId: "100" instead of"1001" , gets an empty result, and hallucinates around it. - Bad ordering. The agent refunds before verifying the order exists, because process_refund was "easier" to call first. - Inefficient or looping trajectories. The agent calls the same tool repeatedly with identical arguments, burning tokens and latency. - Bad synthesis. Every tool call succeeded, but the final answer contradicts what the tools returned. Notice that only the last of these is visible if you only test the final response. This is why the broader agent evaluation field has converged on a layered approach: use deterministic, code-based graders for everything that is objectively checkable (tool names, arguments, call order, error rates), and reserve LLM-as-a-judge graders for semantic questions (was this tool choice appropriate? is the answer helpful?). Deterministic checks are free, instant, and reproducible, so they should carry as much of your suite as possible. The example agent in this guide is deliberately small, but the failure modes above are exactly what benchmarks like τ-bench and τ²-bench measure at scale: tool-using agents navigating realistic customer-support scenarios. Prerequisites Before you begin, ensure you have: - Node.js: Version 22 or later. Download from nodejs.org. - OpenAI API key: The agent and the LLM judges both use OpenAI models through Mastra's model router. Create a key at platform.openai.com. Any other provider works too; just change the model strings. Set up the project Create a new directory and initialize it: mkdir support-agent-evals && cd support-agent-evals npm init -y && npm pkg set type=module Install the dependencies: npm install @mastra/core @mastra/evals zod npm install --save-dev vitest tsx - @mastra/core : Mastra's core package, which includes agents, tools, and therunEvals evaluation pipeline. - @mastra/evals : The evals package, which includes Quick Checks and all built-in scorers. - vitest : The test runner you'll use to run evals in CI. Any ESM-compatible runner (Jest, Mocha) works. Create a .env file with your OpenAI key: OPENAI_API_KEY=your_openai_key_here Build the agent under test The agent you'll test is a customer support agent for an online store. It has three tools: look up an order, process a refund, and escalate to a human. A refund requires looking up the order first, a realistic business rule that gives you something meaningful to test. Create a src/agent.ts file: import { Agent } from '@mastra/core/agent' import { createTool } from '@mastra/core/tools' import { z } from 'zod' // In-memory "database" of orders const orders: Record = { '1001': { id: '1001', status: 'delivered', total: 59.99 }, '1002': { id: '1002', status: 'shipped', total: 129.0 }, '1003': { id: '1003', status: 'processing', total: 24.5 }, } const refundedOrders = new Set () export const lookupOrderTool = createTool({ id: 'lookup_order', description: 'Look up an order by its ID to get status and total', inputSchema: z.object({ orderId: z.string().describe('The order ID, e.g. "1001"'), }), outputSchema: z.object({ found: z.boolean(), order: z.object({ id: z.string(), status: z.string(), total: z.number() }).optional(), }), execute: async ({ orderId }) => { const order = orders[orderId] return { found: Boolean(order), order } }, }) export const processRefundTool = createTool({ id: 'process_refund', description: 'Process a refund for a delivered order. Always look up the order first.', inputSchema: z.object({ orderId: z.string().describe('The order ID to refund'), }), outputSchema: z.object({ refunded: z.boolean(), reason: z.string().optional(), }), execute: async ({ orderId }) => { const order = orders[orderId] if (!order) return { refunded: false, reason: 'order_not_found' } if (order.status !== 'delivered') return { refunded: false, reason: 'order_not_delivered' } if (refundedOrders.has(orderId)) return { refunded: false, reason: 'already_refunded' } refundedOrders.add(orderId) return { refunded: true } }, }) export const escalateTool = createTool({ id: 'escalate_to_human', description: 'Escalate to a human agent when the customer is upset or the request is outside policy', inputSchema: z.object({ reason: z.string().describe('Why the conversation is being escalated'), }), outputSchema: z.object({ escalated: z.boolean(), ticketId: z.string(), }), execute: async ({ reason }) => { return { escalated: true, ticketId: TICKET-${Math.floor(Math.random() * 10000)} } }, }) export const supportAgent = new Agent