Testing AI agents is hard. I built a framework for it.

Your AI agent works in dev. You change a prompt to improve tone. Now it stops routing billing questions correctly. You don't find out until a user complains.

The problem: AI agents are non-deterministic. Traditional unit tests don't work. expect(output).toBe("transfer to billing") fails 30% of the time on correct behavior, and passes when the agent is broken in subtle ways.

I built AgentSpec to fix this - a testing framework designed specifically for non-deterministic AI output.

What it looks like

Define tests in YAML:

name: "billing-agent-tests"
tests:
  - name: "routes billing question"
    input: "I want a refund"
    expect:
      contains_any: ["billing", "refund", "support"]
      tool_called: "transfer_to_billing"
  - name: "handles expired token"
    input: "my token expired"
    expect:
      contains: "refresh token"
      not_contains: "I don't know"
      max_latency_ms: 5000
  - name: "response is on-topic"
    input: "how do I reset my password?"
    expect:
      semantically_similar: "reset password credentials"
      min_confidence: 0.5

Run them:

npm install -g @ozperium/agentspec
agentspec init
agentspec run

Output:

AgentSpec v1.2.0
✓ routes billing question (312ms)
✓ handles expired token (891ms)
✗ response is on-topic
Expected semantic similarity ≥ 0.5, got 0.31
Input: "how do I reset my password?"
Output: "Please contact support for account issues."

2 passed, 1 failed

Why assertions built for AI

Standard test assertions assume deterministic output. AI output isn't. AgentSpec provides:

• contains / not_contains - substring presence (handles paraphrasing)
• contains_any / contains_all - flexible multi-keyword matching
• semantically_similar - word-overlap similarity score, not exact match
• regex - pattern matching for structured output
• tool_called - verify the agent invoked the right tool
• max_latency_ms - catch regressions in response time
• llm_judge - use a local model (Ollama) to evaluate quality in natural language

CI integration

agentspec run --ci   # exits 1 on failure, JUnit XML output

Drop it in GitHub Actions:

- name: Run AgentSpec
  run: agentspec run --ci

Now every PR that changes a prompt, model, or tool gets behavior regression tests.

Behavior diff reports

Run with --diff to compare against a previous baseline:

agentspec run --diff baseline.json

Shows exactly what changed between runs - useful when you swap models or update prompts.

HTTP agents

Point it at any running agent endpoint:

agentspec run --endpoint http://localhost:3000/chat

No SDK integration required.

Install

npm install -g @ozperium/agentspec
agentspec init
agentspec run

GitHub: https://github.com/Ozperium/agentspec

Also in the stack: AICostTracker for tracking what you spend on AI APIs, and quota for monitoring rate limits before they stop you.