12 questions to ask before you buy a text-to-SQL tool

Every demo works. The demo is the easy part. These are the questions that separate a tool your finance team can rely on from one that quietly produces wrong numbers in a nice font. Natural-language querying demos beautifully. Someone types show me revenue by region last quarter, a chart appears in about a second, and everyone in the room can immediately think of ten questions they'd ask it. That reaction is real, and it's why this category is growing. It's also why evaluations go badly. The demo tests the one thing that is no longer hard - a modern model writing plausible SQL against a small, clean schema. It tests none of the things that decide whether the tool survives its first quarter: whether the number is right, whether the person asking was allowed to see it, and what happens when a generated query meets a production database at month-end. Below are twelve questions, grouped by the kind of failure they catch. For each one there's the answer you want, the answer that should slow you down, and a way to test it inside the meeting rather than six months into a contract. Ask them of every vendor, including us. Question 01 What is your accuracy, and how did you measure it? This is the single most useful question in the list, because almost nobody in the category can answer it and the reaction tells you everything. Text-to-SQL accuracy is respectable on small, clean, well-documented schemas and drops sharply on real ones - hundreds of tables, cryptic column names, dirty values, ambiguous join paths. The industry benchmark BIRD exists precisely because results on tidy academic schemas did not survive contact with production databases. Whatever a vendor's number is, some share of answers will be wrong, and the users you're buying this for cannot tell which ones. Good answer A specific figure, the size and source of the question set it was measured on, and an offer to re-run it against your schema during a pilot. Bonus if it runs automatically on every model and prompt change. Warning sign "It uses GPT-class models, so it's very accurate." Model quality is an input, not a measurement. Also treat "we've never had a complaint" as what it is - evidence that nobody checked. Test it live Bring 10 questions you already know the answers to. Ask for all 10 in one session. Count. Question 02 What exactly does "verified" mean in your product? Products in this space like verification badges, confidence indicators and green check marks. Ask what the check mark is asserting. Usually it means the query parsed and executed without an error - which is a statement about SQL validity, not about whether the answer is true. A query can execute perfectly and still double-count revenue because an order joins to three line items. There is no error to catch. The chart looks fine. Good answer A clear distinction between "this ran" and "this matches a known-correct result," plus a way for a human to mark a query as reviewed and reuse it by name afterwards. Warning sign The badge turns out to mean "no exception was thrown," or the vendor can't say what it checks. Test it live Ask a question whose correct answer requires a distinct count. See whether it fans out - and whether anything flags it. Question 03 If two people ask the same question differently, do they get the same number? "Revenue" is not a column. Gross or net of refunds? Booked or shipped? Tax included? Recognized when? Each is defensible, and a model picks one per query based on phrasing. The failure mode is organizational rather than technical: two people bring different numbers to the same meeting, both produced by the tool, and trust in the whole system dies in that meeting. No prompt fixes this, because the ambiguity lives in your business, not in the model. Good answer A semantic layer - metrics defined once, with their join path, filters and grain fixed - so the model selects a defined metric rather than inventing the arithmetic each time. Warning sign "The model figures it out from context," or a suggestion that you write more specific questions. That pushes the definitional work onto every user, every time. Test it live Ask for the same metric three ways in one session. Compare all three numbers. Part two - questions 4 to 6 Access: who is actually asking Most tools have permissions. Fewer have permissions the model cannot route around. Question 04 Is row-level access enforced in the database or in your application? This is the question that most often changes a decision, and the wording matters - almost every vendor says "granular permissions," and that phrase covers two very different architectures. If access is filtered in the application layer, then a generated query still runs against everything and the product decides what to show you afterwards. If it's enforced by row-level security in the database, the query physically cannot return rows the person isn't entitled to, no matter what SQL the model produced. Good answer Row-level security policies in the database, with the asking user's identity set on the connection for each request. The vendor should be able to show you the policy. Warning sign "We instruct the model to only query the user's own data." A prompt instruction is a suggestion. Also worrying: one shared read-only connection for the whole workspace. Test it live In a pilot, log in as a restricted user and ask for something out of scope. Then check the query logs. Question 05 What is the tool allowed to write, and what enforces that bound? "Read-only by default" is now standard phrasing, and the important word is default. Most products in this category also offer actions - alerts, webhooks, CRM updates, scheduled jobs - and those are writes. So the real question is what holds once you've turned on the features you're buying it for. Good answer Read-only enforced by the database role's own privileges, so it holds regardless of what the model emits or what someone toggles in settings. Writes, if any, go through a separate, narrowly-scoped path. Warning sign The only bound is a setting in the product, or a human approving an action they'd need to read SQL to evaluate. Test it live Ask them to connect with a role that has write privileges revoked, and see whether the product still works. Question 06 How do you stop generated SQL from doing something you didn't intend? Models don't reliably follow rules written in prompts, so enforcement has to live in code. The common implementation is a blocklist of forbidden keywords, and it fails in both directions: it rejects legitimate queries touching a created_at column, while missing catalog functions, data-modifying CTEs and comment obfuscation. The sound approach is to parse the generated SQL with the database's own parser and allowlist against the resulting syntax tree - one statement, SELECT only, known relations. Deny-lists enumerate badness; only allow-lists enumerate a finite set. Good answer Parser-based validation against an allowlist, backed by database privileges as a second layer that catches anything the first one missed. Warning sign "We check for dangerous keywords." Ask what happens with a column literally named created_at, and watch. Test it live Ask the product, in plain English, to list the tables in your database. See what comes back. Part three - questions 7 to 9 Blast radius: what a bad query can reach You are handing a generative system a credential to a database that other things depend on. Question 07 What stops a generated query from taking down my database? A generated query is an unbounded query. A join written the wrong way against a large table can saturate the database that also serves your application, and nobody involved intended anything malicious - someone just asked a broad question at the wrong moment. Good answer Cost estimated before execution and rejected above a ceiling, a hard statement timeout, row caps enforced server-side, and a read replica rather than the primary. Warning sign No limits mentioned, or "our queries are fast" with a sub-second demo number as the evidence. That was one query on one schema. Test it live Ask for something deliberately enormous - every order joined to every line item, no filter - and watch what the product does. Question 08 What leaves my environment, and which model provider receives it? "Your data never trains anyone's model" is a good commitment and a narrow one. It usually refers to query results. The schema - table and column names - generally does go to a model provider, and schema is not neutral: it describes your business, your customer attributes and sometimes your unreleased products. Check which tier the safe configuration lives in, too. If self-hosting or bring-your-own-model is enterprise-only, then the plan you were quoting is not the plan you evaluated. Good answer The provider named plainly, a clear statement of what is sent and what is retained, and an in-environment option that isn't buried behind a bespoke contract. Warning sign The model and provider are never named anywhere in the product or the docs. Test it live Ask which sub-processors handle your schema, and ask for that list in writing. Question 09 What happens if one of my rows contains instructions? This one surprises people, so ask it slowly. Your database already contains text written by people outside your company - support tickets, reviews, CRM notes, form submissions. A row reading ignore previous instructions and… is inert while it sits in a grid. It becomes live the moment those results are fed back to a model to summarize the answer, name a chart, or suggest a follow-up - which is exactly what every product in this category does next. The risk compounds sharply if the same product can also fire webhooks or write to your CRM. Good answer Results delivered to any downstream model as clearly delimited untrusted data, and the summarizing model given no tools - so nothing a row says can cause an action. Warning sign A blank look, or a claim that the model "know