Agentic reliability and evaluations : Enterprises that got burned by a bad eval are the most likely to remove humans from the loop, not the least

Across 108 enterprises, trust in automated agent evaluation rose sharply in July - and the failure rate it is supposed to predict did not move at all. The share of organizations that fully trust automated evaluation nearly tripled, from 5% in June to 13%, and the complaint that evaluations don’t match real-world outcomes fell 10 points. Yet the same share as last month - just under half - shipped an agent that passed its evals and then failed a customer. The reason is visible in the cross-tabs: the new trust belongs almost entirely to enterprises that have not yet been burned. Among those that have, 4% trust automated evaluation; among those that haven’t, 24% do. And getting burned does not slow the march to autonomy - it speeds it up. This is the second wave of the VentureBeat Pulse Research agent reliability tracker, and the first fielded on an instrument identical to the month before it. That makes July the first read on direction rather than position: what moved, what held, and what the movement means. What moved is confidence. In June, only 5% of enterprises said they fully trusted automated evaluation, and the most-cited limitation was that evaluations align poorly with real-world outcomes (29%). In July, 13% fully trust automated evaluation and the alignment complaint has fallen to 19%, no longer the leading objection. Both shifts are large enough to read as real rather than noise. What held is the failure. Just under half of organizations (49%) deployed an agent or LLM feature in the past year that passed internal evaluations and then caused a customer-facing failure - statistically indistinguishable from June’s 50% - and a quarter (24%) have seen it happen more than once. Confidence improved; correctness did not. That is the July gap: not between autonomy and trust, as in June, but between trust and the evidence for it. The cross-tabs explain where the new confidence comes from, and it is not from better evaluations. Trust is concentrated almost entirely among enterprises that have not experienced a false-confidence failure: 24% of them fully trust automated evaluation, against 4% of those that have. The trust curve is being lifted by inexperience. Meanwhile the enterprises that have been burned are not retreating from autonomy - 85% of them already allow zero-human deployment or are engineering toward it, against 61% of those that have not been burned. Overall the autonomy trajectory is flat at 67%, but the population inside it has shifted toward the organizations with the most direct evidence that evaluations miss things. The vendor market, by contrast, is finally showing signs of settling. The share of enterprises running no dedicated evaluation tooling fell from 17% to 12%; specialist platforms gained, with Braintrust nearly doubling to 15% and DeepEval reaching 17%; and switching intent cooled, with those planning no change rising from 36% to 44%. Selection criteria moved with it: ease of integration overtook cost as the top factor, jumping from 27% to 39%. Enterprises are done shopping on price and have started buying on fit. Methodology VentureBeat fielded this survey as part of its ongoing Pulse Research series. This wave - the agentic reliability and evals tracker - examines how technical leaders evaluate agent performance and reliability. Responses are filtered to organizations with 100 or more employees (n=108), drawn from a July 2026 fielding. Because the July instrument is identical to June’s, this report makes month-over-month comparisons where they are warranted; where questions were multiple-select, shares can sum to more than 100%. Comparisons against June (n=157) are tested for significance, and only a handful of the month’s movements clear a conventional threshold: the rise in full trust in automated evaluation (5% to 13%), the fall in the real-world-alignment complaint (29% to 19%), the jump in ease of integration as a selection factor (27% to 39%), and the gain in Braintrust as a primary platform (8% to 15%). Movements described in this report as flat - the failure rate, the autonomy trajectory, the production monitoring mix, the investment ranking - are statistically indistinguishable between waves, and that stability is itself the finding. Differences of a few points elsewhere should be read as sample variation, not trend. By role the sample is senior and buyer-credible: 44% are final decision-makers for AI purchases and another 25% recommenders or influencers, a slightly more senior mix than June. Product and program managers (18%), consultants and advisors (12%), CIOs/CTOs/CISOs (11%), and directors of engineering/IT (11%) lead the named titles, alongside a large “Other” function (30%). By organization size the sample is again mid-market-weighted: 100-499 (33%) and 500-2,499 (30%) employees lead, with 2,500-9,999 (23%), 10,000-49,999 (9%), and 50,000+ (5%) above them. One composition change is worth flagging because it bears on the trust finding. The industry mix shifted between waves: Technology/Software fell from 23% of the June sample to 14% in July, while Retail/Consumer rose from 15% to 19% and now leads. A less technology-weighted sample plausibly carries less hands-on exposure to agent evaluation, and some of the month’s rise in trust may reflect who answered rather than what changed. The burned-versus-unburned split reported in Finding 2 holds within the July sample regardless, but readers should treat the headline trust movement as directional. At 108 respondents the sample is large enough to support directional conclusions but should not be treated as a precise measurement; it is self-selected and is not a probability sample. Cross-tabs reported here rest on subgroups of 40 to 68 respondents and are correspondingly coarse. Finding 1: The failure rate did not move Just under half still ship agents that pass evals and fail customers We asked whether, in the past 12 months, organizations had deployed an agent or LLM feature that passed their internal evaluations but then caused a customer-facing failure. The answer is the same as last month. Finding 1 - The Failure Rate Did Not Move Forty-nine percent of organizations shipped an AI feature that cleared internal evaluations and then failed in front of a customer - an incorrect output, a broken workflow, or a quality incident - against 50% in June. A quarter (24%) have seen it happen more than once, unchanged. Across two waves and 265 enterprises, the rate at which evaluations certify agents that then fail is stable to within a percentage point. That stability is the anchor for everything that follows. Every other movement this month - rising trust, consolidating tooling, shifting purchase criteria - has to be read against a failure rate that has not responded. Whatever enterprises did between June and July, it did not change how often a passing evaluation turns out to be wrong. Finding 2: Trust rose - among those who haven’t been burned Full trust nearly tripled, and the alignment complaint fell ten points We asked which limitation most reduces trust in automated agent evaluations today. The distribution shifted materially from June. Finding 2 - Trust Rose - Among Those Who Haven’t Been Burned Two things moved together: Full trust in automated evaluation nearly tripled, from 5% to 13%, and the objection that most directly describes a false-confidence failure - poor alignment with real-world outcomes - fell from 29% to 19%, surrendering the top spot to evaluation bias and inconsistency (22%), now tied with data-leakage concerns (22%). On the surface this reads as an evaluation layer beginning to earn its keep. The cross-tab says otherwise. Splitting the sample by whether an organization has actually experienced a false-confidence failure, trust divides almost completely. Among the 53 enterprises that shipped an agent which passed evals and then failed a customer, 4% fully trust automated evaluation. Among the 41 that have had no such failure, 24% do - a six-fold difference, and the sharpest split in the dataset. Direct contact with the failure mode is what removes the trust. This is the month’s central caution. The improvement in sentiment is not evidence that evaluations got better; the failure rate in Finding 1 rules that out. It is what a trust curve looks like when a cohort of less-burned organizations enters the sample and reports its priors. Enterprises reading their own rising confidence as validation of their evaluation stack are reading a number that measures inexperience. Finding 3: Being burned accelerates autonomy rather than restraining it 85% of the burned are on the zero-human path, against 61% of the REST We asked whether organizations would let an autonomous agent deploy a code or system change to production on automated evaluation results alone, with no human-in-the-loop validation. The aggregate held; the composition did not. Finding 3 - Being Burned Accelerates Autonomy Rather Than Restraining It At the top line, nothing changed: 67% of organizations either already allow zero-human-in-the-loop deployment for low-risk agents (37%) or are actively engineering their pipelines to permit it within a year (30%), against 67% in June. The share ruling it out for the foreseeable future slipped from 22% to 18%. The autonomy ceiling stopped rising, but it did not come down. Underneath, the picture inverts the intuitive one. Among enterprises that have shipped an evaluation-passing agent that then failed a customer, 85% are on the autonomy trajectory. Among those that have not, 61% are. Organizations with direct, expensive evidence that their evaluations miss things are substantially more likely to be removing the human check, not less - and only 11% of them rule out full automation, against 24% of those that haven't been burned. The pattern is identical for those burned once and those burned repeatedly. The most plausible mechanism is not recklessness but maturity: the organizations that ship agents at enough vo