Testing AI Agents: Why Unit Tests Aren't Enough
DEV Community

Testing AI Agents: Why Unit Tests Aren't Enough

You ship a change to your agent's system prompt. You run the tests. They pass. You deploy. Three days later, your support agent starts filing refund requests as generic support tickets.

The prompts look fine. The code looks fine. Nothing broke - except the part that matters.

This is the testing gap in AI systems. Traditional unit tests are still essential, but they don't tell you whether your agent behaves correctly across the range of inputs your users will send. You need something else: evaluation tests, or evals.

Why Unit Tests Aren't Enough

A unit test checks deterministic behavior:

expect(parse_date("march 20", today: Date.new(2026, 1, 15))).to eq("2026-03-20")

Call it a thousand times, get the same result a thousand times. Any deviation is a bug. (Note the pinned today: - without it, even this test changes answer on January 1st.)

An LLM call is different. It's a probabilistic system. Given the same input, it generates a likely output - not necessarily an identical one. Sampling parameters influence this behavior, and even with temperature: 0, hosted models can produce slightly different results between runs.

The naive approach - run the agent and assert its output - fails in two ways:

  • False failures: The output is semantically correct but formatted slightly differently. Your test breaks on a Tuesday.
  • False passes: The output passes your specific assertion, but the model has regressed on cases you never tested.

Unit tests can verify the code around an LLM. They cannot, by themselves, establish that the model behaves correctly. You have to evaluate it.

What You're Actually Testing

When an agent processes a message, there are three distinct things that can go wrong:

  1. The noop decision. Did the agent correctly decide to act - or not act? "cn u snd towles to rom 305 thx" → should trigger an action. "👍" → should not. Getting this wrong is expensive: false positives flood your backend with phantom events, while false negatives silently drop real requests.
  2. The action type. Did the agent pick the right action from the allowed list? "My order ORD-8832 arrived damaged" should produce support.refund.process, not support.ticket.create. Both are valid action types. Only one is correct.
  3. Field extraction. Did the agent extract the right information from the message? booking_ref: "AB1234" new_date: "2026-03-20" These are the values your backend acts on. A wrong date changes a booking to the wrong day. A missing order ID stalls a refund.

A single pass/fail assertion can't tell you which of these three failed.

The Eval Dataset

An eval suite starts with a labeled dataset. Each case represents a realistic input with an explicit expected output:

  • id: h05
  • tags: [typo]
  • message: "i nee to chaneg my bokking to april 3rd plz ref XY789"
  • expected:
    • action_type: hospitality.booking.modify
    • noop: false
    • fields:
      • booking_ref: "XY789"
      • new_date: "2026-04-03"

One detail that bites later: "april 3rd" only means 2026-04-03 relative to a reference date. Pin that date and pass it to the agent on every run, or your expected values silently rot when the calendar moves.

The tags are important. You don't just want a pass rate - you want to know where the model struggles. For example:

  • clean: Well-formed messages - the baseline
  • typo: Spelling errors, autocorrect, missing vowels
  • emoji: Heavy emoji usage
  • multilang: French, Spanish, German mixed with English
  • ambiguous: Multiple valid interpretations
  • implicit: Intent not stated directly
  • multi_intent: Two requests in one message
  • noop: Should produce no action
  • context: Follow-up requiring previous conversation state

The ambiguous and implicit tags are often where gaps appear first. "The room is too cold" - is this a room service request, or just a complaint? (ambiguous) "I don't think I'll make it" - is this a cancellation? (implicit)

Clean examples almost always pass. The interesting failures are the ones that look like real user messages.

The Scoring Model

The runner iterates over every case, calls the agent, and evaluates its output across three axes. For repeatability, we use temperature: 0. This reduces sampling variability, but does not guarantee identical outputs across runs, especially when using hosted APIs.

Here's a simplified evaluator:

def evaluate(expected, output)
  errors = []
  # Axis 1: noop detection
  expected_noop = expected["noop"] == true
  actual_noop = noop_output?(output)
  unless expected_noop == actual_noop
    errors << "noop: expected=#{expected_noop}, got=#{actual_noop}"
  end
  # Axis 2: action type
  unless expected_noop
    expected_type = expected["action_type"]
    actual_type = extract_action_type(output)
    unless expected_type == actual_type
      errors << "action_type: expected=#{expected_type}, got=#{actual_type}"
    end
  end
  # Axis 3: field presence
  if expected["fields"]
    check_fields(expected["fields"], output, errors)
  end
  { pass: errors.empty?, errors: errors }
end

def check_fields(expected_fields, output, errors)
  actual = output.fetch("fields", {})
  expected_fields.each_key do |name|
    value = actual[name]
    errors << "fields.#{name}: missing" if value.nil? || value.to_s.strip.empty?
  end
end

Notice what we don't do here: exact value matching on every field. We start by checking field presence. Why? Because different representations can express the same information. For example, "march 20" and "2026-03-20" may represent the same date. Exact equality would require normalization rules for each field type.

But there's an important trade-off. A presence check won't catch a wrong date. It only catches missing information. For fields that directly influence business operations - dates, amounts, IDs - value validation is essential. A practical approach is to start with presence checks, then add field-specific validation and normalization as the system evolves.

The goal isn't to make every assertion complicated. It's to catch the failures that matter.

Why not use an LLM as a judge? Because the output here is structured: a noop flag, an action type, a set of fields. Deterministic checks are free, fast, and don't add a second source of variance. A judge earns its place when the output is free text.

Baselines and Regression Detection

After each run, save the results to a timestamped file:

spec/eval/results/
  20260307_baseline.yml
  20260308_224748_support_recheck.yml
  20260309_122811_gemini-2.0-flash.yml

The label matters. You're not looking at a single score - you're tracking a trajectory. (All runs in this article date from March 2026; model names and prices reflect that period.)

For example:

# 20260307_baseline.yml
scores:
  action_type: 95.8
  fields: 85.7

These scores become your behavioral baseline. Break them down by tag too: a global score hides most of the signal, and a drop on multilang alone tells you far more than a one-point dip overall.

Improve the system prompt? Run the eval. Scores hold or improve → good sign. Scores drop significantly, or a category that previously scored 100% starts failing → investigate.

Don't treat every small variation as a regression. Hosted models exhibit run-to-run variance, so measure it: run the same configuration a few times and note how much the scores move on their own. Anything inside that spread is noise.

The key discipline: Always run the eval before merging a change that affects model behavior. A diff on a YAML prompt file won't tell you if you just broke field extraction on multilingual inputs. CI integration is useful, but optional. Running the suite consistently matters more than automating it immediately.

What the Failures Teach You

The first baseline surfaced 10 failures. The patterns were instructive.

Ambiguous cases fail predictably. "The room is too cold" produced no action at all. The model wasn't confident enough to commit to hospitality.room_service.request without more signals. This isn't necessarily a bug. It's a calibration question: do you want confident-only extractions or best-effort behavior?

Fields fail on implicit mentions. "✅ driver karim delivered everything in nice 🏁" The model got the action type right (operations.driver.update) but missed destination: Nice. "Nice" is a city, but also an adjective. Field extraction on ambiguous proper nouns is genuinely hard.

Cross-domain routing underperforms. "Cancel my booking ref HT9921 and also where is my order ORD-5500?" The expected behavior is to route to both the hospitality and the support domains. The model picked one. Multi-intent cross-domain routing is a difficult case: a single model call has to recognize and preserve multiple intents.

These failures tell you exactly what to work on next - and give you a metric to validate whether your fix actually worked.

As the system evolves, the dataset should evolve too. Add new cases from production failures, edge cases and unexpected user behavior. Otherwise, your eval suite gradually becomes a collection of examples the model already knows how to handle.

The Cost Argument

Here's what makes this approach sustainable: 98 cases × gpt-4o-mini = $0.0106

Just over one cent for a full behavioral evaluation suite, based on the model and pricing used for this run. You can run it after every prompt change, every model swap, or every meaningful pipeline update.

Want to evaluate a different model - say, switching from gpt-4o-mini to gemini-2.0-flash? Run the same dataset against both and compare:

# gpt-4o-mini
scores:
  action_type: 95.8
  fields: 85.7
# gemini-2.0-flash
scores:
  action_type: 94.1
  fields: 83.2

The honest reading: on this dataset, a 1.7-point gap on action type is one or two cases - well within run-to-run variance. These two models are equivalent here, so the decision comes down to cost and latency. That's still a useful answer, and one you couldn't get from a gut feeling.

The exact cost will depend on model pricing, input size and output tokens, but the point remains: a small, focused eval suite can be surprisingly cheap.

Unit Tests, Evals and Integration Tests

These three testing approaches solve different problems. Unit tests verify that your deterministic code works: parsing, validation, permissions and business logic. Evals measure whether the model produces the expected behavior across a representative dataset. Integration tests verify that the generated output works correctly within the rest of your system.

None replaces the others. A correctly classified request doesn't guarantee that the resulting action will execute successfully. Likewise, a passing integration test doesn't prove that the model will classify the next 100 user messages correctly.

The Pattern Distilled, the approach looks like this:

  • Build a dataset of realistic inputs with tagged categories
  • Define scoring dimensions, not just pass/fail
  • Save results after every significant change
  • Protect the baseline and measure run-to-run variance
  • Add new cases from real failures and edge cases
  • Use the results to prioritize what to fix next

No framework required. It's a dataset, a runner, and a disciplined habit of checking model behavior before shipping changes.

Traditional tests tell you whether your code works. Evals tell you whether your AI system does what you expect. The teams that skip this often discover regressions only after a user reports them. By then, you don't know which change caused the problem, and you're debugging blind.

If your AI system is important enough to ship, its behavior is important enough to test.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.