DEV Community 1h ago

Your New Prompt 'Feels' Better. That's Not an Eval.

You tweak the prompt. Run it against the three examples you always use to sanity-check. It looks better. Ship it.

That's not evaluation. That's vibes with extra steps - and it's the exact habit interviewers are trained to catch with one question.

TL;DR

One interview question, one real framework: don't trust "it looks better on my usual examples" - build a small eval set, score it the same way every time, and re-run it on every future change. Read the framework, then use it as the literal answer next time someone asks.

The Question

"How do you know your new prompt is actually better - not just better on the five examples you tried by hand?"

This is one of the fastest ways an interviewer separates "I iterate on prompts" from "I've actually shipped prompt changes to production." Everyone iterates. Almost nobody evaluates.

How to Answer It

Name the trap first, out loud: eyeballing a handful of hand-picked examples is confirmation bias, not evaluation. You'll always find a few cases where the new version looks better - that's not evidence, that's cherry-picking with good intentions.
Describe a minimal eval set: 20-50 real or realistic input/expected-output pairs, run automatically against both the old and new prompt, scored the same way every time - not re-read by eye each round.
Say what "scored" actually means, because this is where most candidates go vague: exact match works for structured output (JSON fields, classifications); for open-ended text, you need a similarity or grounding score against a reference answer.
Close with the regression angle: the eval set isn't a one-time gate before this ship - it's something you re-run on every future prompt or retrieval change, so a "small tweak" three weeks from now doesn't quietly break something the old version got right.

Why "It Looked Better to Me" Is Losing You the Room

Here's the uncomfortable part: this question isn't really testing whether you know what an eval set is. It's testing whether you've been burned by not having one - shipped a "better" prompt that regressed silently, found out from a user complaint instead of a dashboard.

Candidates who've lived that answer differently than candidates who haven't, and interviewers can usually tell within one follow-up question. Somewhere in your interview pool right now, another candidate already built that eval habit into their process - not because a course told them to, but because something broke on them once. That's the gap this question is actually measuring.

What the Eval Set Looks Like When It's Not Hypothetical

Skip the abstraction. In practice it's a spreadsheet or a JSON file with columns:

input
expected/reference answer
prompt-A output and score
prompt-B output and score

Twenty rows is enough to start. The moment you can point to a number instead of a feeling, you've already answered this question better than most candidates do.

Next: The 12 Lines of Code That Do the Scoring

Saying "similarity score" in an interview is good. Being able to sketch the actual code is better - and next week's post is exactly that: a Colab-ready snippet that scores whether a generated answer is actually grounded in retrieved context, the same technique behind the eval-set scoring above.

The takeaway: "it looks better" is an opinion. A score you can re-run is an answer.

See the paths that build that judgment →

Preparing for AI Interviews?

Don't prepare alone. Join a live session where we break down real interview questions like this one, discuss how strong candidates structure their answers, and work through the follow-up questions interviewers actually ask. Bring your questions, test your answers, and learn live with other candidates preparing for AI roles.

👉 Join the next live AI interview prep session →