How to Add Evals to an LLM Feature

Why Evals Are Not Optional

LLMs are non‑deterministic: give them the same input twice, and you’ll get two different responses. That means unit tests that check for exact string matches are useless. As Pragmatic Engineer notes, you need evals to verify that the solution works well enough - because there’s no guarantee it will.

When you’re building a feature that speaks to real customers, like the AI Calling Agent dashboard we built, a regression in tone or missed booking intent can cost revenue immediately. Evals turn that uncertainty into signal.

How to Add Evals to an LLM Feature: A 4‑Step Workflow

We’ll walk through the exact process we followed, from defining success to automating checks in CI, using the DeepEval framework as an example. You can swap in Evidently AI or build your own, but the pattern is the same.

Step 1: Define Success for Your Feature

Takeaway: Before you pick a metric, write down the one thing that makes the feature “done” - usually a business outcome, not a technical measure.

For the AI Calling Agent, the core feature was an outbound call that books a meeting. The success criterion wasn’t “the LLM replied politely.” It was “the agent scheduled a meeting with the right time and date.” This is a reference‑based evaluation: you compare the output to a known ground truth. Evidently AI’s guide calls this pattern out as essential for regression testing and experimentation.

From that criterion, we derived a concrete metric: successful_booking - a boolean that checks whether the transcript contains a confirmed appointment. Later, we layered on softer metrics like tone_appropriateness and objection_handling.

# definition of our primary metric
from deepeval.metrics import GEval

booking_metric = GEval(
    name="successful_booking",
    criteria="Determine whether the agent successfully booked a meeting with the correct date and time.",
    evaluation_steps=[
        "Check if the transcript contains a confirmed appointment.",
        "Extract the date and time mentioned.",
        "Verify that the date and time are valid and not contradicted by the user."
    ],
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],
)

Step 2: Build a Representative Eval Dataset

Takeaway: Your eval dataset is the spec. If it doesn’t cover the real failures you’ve seen, your eval will pass - and the feature will still fail.

We started with 20 transcripts from real test calls: 10 where the agent succeeded, 10 where it failed. For each, we recorded the conversation turn‑by‑turn and the expected outcome. The arXiv practical guide emphasizes that you should proactively curate representative datasets, not just sample randomly.

We included edge cases: strong accents, interruptions, customers who said “call me back later.” We structured the dataset as a list of LLMTestCase objects:

from deepeval.test_case import LLMTestCase

test_cases = [
    LLMTestCase(
        input="Hi, this is Alex from Acme. I'm calling about your interest in the demo...",
        actual_output="[full transcript]",
        expected_output="meeting_booked",
        context=["customer is interested, available Tuesday 2pm"],
    ),
    # ... 19 more cases
]

Step 3: Choose the Right Metrics and Scorers

Takeaway: Mix LLM‑based scoring with deterministic checks. LLM judges are flexible but can drift; non‑LLM scorers ground your eval.

For the successful_booking metric, we used an LLM judge (GPT‑4o) with a structured extraction prompt. But we also added a second scorer: a Natural Language Inference (NLI) model that classifies whether the transcript “entails” the booking. As Confident AI’s metrics guide explains, NLI scorers are a solid non‑LLM option that can be run cheaply and consistently.

from deepeval.metrics import NLIConflictMetric

nli_metric = NLIConflictMetric(threshold=0.5)

# Evaluate the same case with both scorers
booking_metric.measure(test_case)
nli_metric.measure(test_case)

We then set a threshold: if both scorers agree, the result is trusted; if they disagree, the case is flagged for manual review. This hybrid approach caught regressions that a single LLM judge missed.

Step 4: Automate Evals in Your CI/CD

Takeaway: An eval that only runs in a notebook is a decoration. The real value comes when it blocks a broken release.

We wired the eval suite into a GitHub Action that runs on every pull request to the agent’s prompt configuration. The pipeline fetches the latest model, runs the entire dataset, and fails if the successful_booking rate drops below 90% or the NLI score dips.

# snippet from .github/workflows/evals.yml
- name: Run eval suite
  run: |
    deepeval test run tests/test_booking.py

You can also use DeepEval’s ephemeral AI skill to generate synthetic edge cases and expand your dataset automatically. The point is to make evals as automatic as linting.

Where We Applied This: The AI Calling Agent

When we built the AI Calling Agent dashboard, evals weren’t an afterthought - they were the first thing we instrumented after the voice pipeline. The agent uses LiveKit for real‑time audio, Twilio for telephony, and OpenAI’s Realtime API. Every tweak to the system prompt or the conversation flow could silently degrade booking rates.

We set up a nightly eval job that replays stored transcripts and compares results against the labeled dataset. If a prompt change causes a 2% drop in confirmed bookings, the team gets an alert before a single real call is made. That’s the kind of feedback loop that turns a cool demo into a product ops teams trust.

We’ve since applied the same pattern to RAG‑based knowledge bases, chatbots, and internal tools. If you’re shipping an LLM feature today, our AI services include eval pipeline design as a first‑class deliverable. Start a project and we’ll help you build what you just read - tailored to your stack.

FAQ

What’s the simplest way to start adding evals to my LLM feature?

Start by defining a single quality criterion for your feature (e.g., “the agent booked the appointment”). Then build a small dataset of 10–20 input‑output pairs with ground‑truth labels, pick a scorer like an LLM judge or a classification metric, and run it manually. Once you trust the signal, fold it into CI.

How do I know if my evals are good enough?

LLM‑as‑a‑judge scorers are the most common, but they can be inconsistent. Pair them with non‑LLM scorers like NLI models or structured extraction for stability. The real test is whether your eval catches regressions faster than a user complaint - so run it against known failures.

Can I use LLMs to evaluate other LLMs?

Absolutely. Many eval frameworks use an LLM to judge aspects like correctness, helpfulness, and tone. This is fast and flexible, but it introduces a second layer of non‑determinism. Always validate the judge against a human‑labeled sample before trusting it.