Five Red Flags I Look For in Every AI Tool Demo (That Most Buyers Miss)

Red Flag 1: Handling Questions Outside the Indexed Content

I always ask a question during a demo where the answer is not in the indexed content, framed as a genuine inquiry rather than a test. The tool's behavior here tells you more about its reliability than ten impressive answers on covered topics.

Tools worth considering acknowledge the gap: they say some version of "I don't have reliable information about this in the available documentation." Tools to be skeptical of generate a fluent, confident, plausible-sounding answer anyway. That behavior in a demo is exactly that behavior in production, applied to real employees asking real questions and trusting the outputs.

Red Flag 2: Evasion on the Data Flow Question

At some point in every evaluation, I ask the vendor to describe exactly where my data goes during an AI interaction. Not their security posture. Not their certifications. Specifically: when an employee submits a query, what happens to that query and to the retrieved context? Which servers, which services, what is logged, what is retained?

Vendors who have clear answers to this give them directly. Vendors who do not give clear answers redirect to their enterprise agreement, their SOC 2 certification, or their privacy policy. These are not answers to the question I asked. Redirecting to compliance documentation rather than describing the actual data flow tells you the vendor either does not know or does not want to say. Either is informative.

Red Flag 3: Uniform Confidence Across All Queries

Real knowledge retrieval systems have a distribution of query difficulty. Some questions have clear answers in well-maintained documents. Some questions have ambiguous answers across conflicting documents. Some questions should be declined because the information is not available.

A system that treats all queries the same, outputting equally confident responses regardless of the underlying evidence quality, is a system that is not communicating its actual uncertainty to users. Ask to see the tool respond to an ambiguous question where reasonable documents take different positions. Ask to see it respond to a question where the available documentation is outdated. The variance in how the tool handles these cases relative to easy cases tells you whether it is actually reasoning about evidence quality or just always sounding confident.

Red Flag 4: The Invisible Admin Interface

The administrative experience of an AI tool is completely invisible during most evaluations because evaluations are run from the end user perspective. The admin interface is where access control is actually configured, where usage is monitored, where problematic outputs can be investigated, and where user management happens.

Ask to see the admin interface during the demo, live, not in a recorded walk-through. Ask the vendor to show you specifically how you would identify which documents were retrieved for a given query that produced a problematic output. Ask them to show you how you would revoke a specific user's access to a specific category of content. The ease or difficulty of these tasks in the admin interface predicts how governable the tool will be in production.

Red Flag 5: No Specific Pricing for Your Scenario

You have told them your organization size, your use case, your expected query volume. If they cannot give you a meaningful estimate of what this costs, they either have a pricing model that is too complex to explain honestly or they are avoiding the conversation because the real number would affect your purchase decision. You need to know the real number before, not after, you sign.

None of these are hostile questions. They are questions that vendors with strong products welcome because they know the answers reflect well on what they have built. The vendors who become evasive or uncomfortable are the vendors who know the honest answer does not serve their sales goal. Pay attention to what makes them uncomfortable. That is more informative than what they prepared to show you.