131 Tests, 4 Layers, $00.03/Run: Why I Built My AI Agent Eval Harness First

Originally published on AIdeazz - cross-posted here with canonical link. I shipped a silent failure. My first production AI agent, a simple lead qualification bot for WhatsApp, passed all 27 unit tests. It handled happy paths, edge cases, even adversarial inputs designed to break prompt safety. The agent went live. Two days later, a client forwarded a screenshot: the bot had hallucinated a 15% discount on a product that didn't exist. Revenue impact: $0. But trust impact: significant. The root cause wasn't a bug in my code, nor a prompt injection. It was a subtle shift in the LLM's internal reasoning, a drift that unit tests, by their very nature, cannot detect. This experience led to a hard decision: no new AI agent feature ships without first integrating into my evaluation harness. I now have 131 tests across four distinct layers, costing me $0.03 per full run. This harness isn't a luxury; it's the bedrock of shipping production AI agents with zero VC funding and a single developer. The Fundamental Flaw of Unit Tests for AI Agents Unit tests verify my code. They assert that my_function(input) returns expected_output . For traditional software, this is sufficient. For AI agents, it's dangerously incomplete. An AI agent's core logic isn't deterministic code I wrote; it's emergent behavior from an LLM interacting with tools and external systems. Consider a multi-agent system designed to process customer inquiries. - Router Agent: Classifies incoming messages (e.g., "sales," "support," "technical"). - Sales Agent: Qualifies leads, retrieves product info from a database, generates a personalized offer. - Support Agent: Accesses knowledge base, schedules appointments. A unit test might verify: router_agent.classify("I need a new laptop") returns "sales" . But what if the LLM behind the router agent, after a model update or a subtle change in its training data, starts classifying "I need help configuring my new laptop" as "sales" instead of "support" ? My unit test still passes because the function classify executed without error. The semantic intent changed. This is where an AI agent evaluation harness 131 tests production becomes indispensable. My 4-Layer Evaluation Harness My harness runs on Oracle Cloud Infrastructure (OCI) Functions, triggered by new code commits. Each layer targets a different aspect of agent reliability. Layer 1: Core Functionality (38 Tests) These are the closest to traditional unit tests, but they operate at the agent's public API level. They verify that the agent can perform its primary tasks. - Example: For a lead qualification agent, tests include: - agent.process_message("I need 100 widgets") -> assertsresponse.includes("quote") andresponse.includes("delivery time") . - agent.process_message("Tell me about product X") -> assertsresponse.includes("features of X") . - agent.process_message("I want to speak to a human") -> assertsresponse.includes("transferring to human") . - These tests use specific, deterministic inputs and check for expected keywords or structured outputs. They catch regressions in tool calls, API integrations, or basic prompt adherence. Layer 2: Semantic Intent & Reasoning (52 Tests) This is where the harness starts to diverge significantly from unit testing. These tests focus on whether the agent understands and acts appropriately based on the user's intent, even with varied phrasing. This layer is crucial for catching the "silent failures" I experienced. - Methodology: Each test case has: - user_input_variations : An array of 3-5 semantically similar phrases (e.g., "I want a quote," "How much does it cost?", "Pricing for X"). - expected_intent : A categorical label (e.g., "request_quote"). - expected_action : The tool or internal function the agent should invoke (e.g.,call_pricing_api ). - expected_output_keywords : Keywords that must appear in the final response. - unexpected_output_keywords : Keywords that must not appear. - - Example Test Case: { "name": "Pricing Inquiry - Product A", "user_input_variations": [ "How much is Product A?", "Cost of Product A please.", "Give me a quote for Product A.", "What's the price tag on Product A?" ], "expected_intent": "request_pricing", "expected_action": "call_product_pricing_tool", "expected_output_keywords": ["price", "Product A", "USD"], "unexpected_output_keywords": ["discount", "shipping cost"] } The harness runs each variation through the agent. It then uses a small, fine-tuned classification model (or a separate LLM call with a strict prompt) to verify the expected_intent from the agent's internal logs and theexpected_action from tool call logs. Finally, it checks the final output againstexpected_output_keywords andunexpected_output_keywords . This catches hallucination of discounts or incorrect routing. Layer 3: Robustness & Edge Cases (29 Tests) This layer pushes the agent with malformed inputs, ambiguous requests, and high-load scenarios. - Ambiguity: "I need help." (Should trigger clarification or human transfer). - Missing Info: "Quote for widgets." (Should ask for quantity). - Out-of-Scope: "Tell me a joke." (Should politely decline or redirect). - Long Inputs: Messages exceeding typical character limits. - Rapid-fire: Sending 5 messages in 2 seconds to simulate burst traffic (tested with a dedicated load testing script, not part of the $0.03/run cost). For each of these, the harness asserts specific fallback behaviors or error messages. Layer 4: Safety & Guardrails (12 Tests) These tests focus on preventing harmful or inappropriate responses. - PII Evasion: Inputs designed to trick the agent into revealing personal data. - Harmful Content: Prompts asking for illegal activities or hate speech. - Sensitive Topics: Inputs related to politics, religion, or medical advice (for non-specialized agents). The harness expects a refusal, a redirection, or a canned safety response. I use Groq for its speed in these checks, as the response time is critical for real-time moderation. The Cost: $0.03 Per Full Run Running 131 tests across four layers isn't free, but it's cheap enough to run on every commit. - LLM Invocations: The primary cost driver. I dynamically route between Groq (for speed-critical, simple classification/refusal checks) and Claude 3 Haiku (for more complex reasoning and output generation). - Groq: ~$0.000008 / 1k tokens. - Claude 3 Haiku: ~$0.00025 / 1k input tokens, ~$0.00125 / 1k output tokens. - OCI Functions: Serverless execution for the harness logic. Billed per invocation and GB-second. Negligible for my scale. - OCI Object Storage: Storing test cases, results, and logs. Also negligible. A typical full run involves: - ~100 Claude 3 Haiku calls (avg 200 input tokens, 100 output tokens) = $0.005 + $0.0125 = $0.0175 - ~30 Groq calls (avg 50 input tokens, 20 output tokens) = $0.000012 - OCI Function execution time: <1 second total. Total: ~$0.0175 + negligible. I round up to $0.03 to account for occasional longer responses or additional internal logging. This cost is a fraction of the potential revenue loss or reputational damage from a single silent failure. The Silent Failure It Caught Last month, I was developing a new feature for a customer support agent: dynamic FAQ generation based on user queries. The idea was to use an RAG system to pull relevant knowledge base articles and summarize them. My unit tests passed. The RAG system retrieved correct articles. The summarization prompt worked on isolated examples. I pushed the code. The eval harness ran. Layer 2, Semantic Intent & Reasoning, failed one test: "Query about refund policy." The expected output was a summary of the refund policy. The actual output included a sentence: "Please note, all refunds are subject to a 10% processing fee." This was a hallucination. Our refund policy has no processing fee. The RAG system had correctly retrieved the policy. The summarization LLM (Claude 3 Haiku, at the time) had added this detail, likely from its general training data about refunds, despite the explicit instruction in the prompt to only use provided context. Without the harness, this would have shipped. A customer asking about a refund would have been told about a non-existent fee, leading to confusion, frustration, and a direct support ticket. The $0.03 cost of that eval run saved me a customer interaction and preserved trust. Conclusion Building an AI agent evaluation harness 131 tests production isn't optional for serious AI development. It's a critical infrastructure component that catches emergent failures unit tests cannot. It's the difference between shipping robust, reliable agents and constantly firefighting silent, trust-eroding bugs. My $0.03 per run is the cheapest insurance policy I've ever bought. Frequently Asked Questions Q: How do you manage the test data for 131 tests? Is it all hardcoded JSON? A: The core test cases are JSON files stored in OCI Object Storage. For variations and negative tests, I use a small Python script that programmatically generates additional inputs based on templates, ensuring coverage without manually writing every permutation. Q: What if the LLM itself changes its behavior, making existing tests fail even if my code is correct? A: This is precisely what the harness is designed to detect. If an LLM update causes a test failure, it's a signal to either adjust the prompt (to guide the LLM back to desired behavior) or accept the new behavior and update the test's expected output. It's a continuous calibration process. Q: How do you handle non-deterministic LLM outputs in your assertions? A: I avoid strict string equality. Instead, I use keyword presence/absence checks, regex patterns, and sometimes a secondary, smaller LLM (like Groq) to classify the agent's output against expected intent or sentiment. This allows for variability while ensuring core requirements are met. Q: Is $0.03 per run sustainable for larger teams or more complex agents? A: Yes. The cost scales with LLM usage, not linearly with the number of