Cybersecurity Reviews Have an Evidence Problem

Software developers are accustomed to an uncomfortable fact: passing the tests does not prove that the software is free of bugs. It proves that the software passed the tests. These are not the same proposition. One may write ten thousand unit tests, watch ten thousand little green ticks march obediently across the screen, deploy the application, congratulate oneself on the elegance of the build, and discover six minutes later that a customer in Belgium has entered an apostrophe into a field nobody expected to contain an apostrophe and brought the entire thing to its knees. This is software. We have learned to live with it. Cybersecurity reviews suffer from a related, and rather more troublesome, problem. Suppose I install a security product on a company's systems and twelve months pass without incident. Was the product effective? Perhaps. Or perhaps nobody attempted anything particularly serious. Perhaps somebody did and another security control stopped it. Perhaps an attack succeeded but remained undiscovered. Perhaps the product quietly prevented seventeen attacks and deserves a medal. From the observable fact that nothing happened, we have somehow arrived at the conclusion that the security software worked. There is rather a lot of unexplored territory between those two statements. That is the evidence problem in cybersecurity reviews. The threat comes before the feature list One of the easiest ways to review security software is to count things. Product A has malware protection, ransomware protection, phishing protection, a VPN, a password manager and something involving artificial intelligence. Product B has all of these things except the password manager. Product A therefore wins six features to five. Simple, certainly. Useful, not necessarily. Before evaluating a security product there is a more important question: what is it supposed to protect, and from what? A password manager, an antivirus application, an endpoint detection platform, an external attack-surface management system and a penetration-testing service can all be placed under the enormous heading of cybersecurity, but they are solving quite different problems. Think of a building. A lock attempts to prevent somebody entering. An alarm attempts to detect somebody entering. A security inspection tries to discover how somebody could enter. A security guard responds when somebody actually does. It would be peculiar to review these things by counting their features and declaring the alarm superior to the lock because it comes with a mobile application. Yet software comparisons routinely do something rather similar. At The Tool Money Lab, our cybersecurity research therefore begins with the threat model. Who is being protected? What is being protected? Against what sort of threat? At which stage of an attack does the product operate? What happens after it discovers something? Until those questions have answers, most feature lists are little more than decoration. A vendor claim is evidence of a claim Developers will recognise another important distinction. Suppose the documentation for a library says that it performs an operation in 20 milliseconds. You may accurately report that the documentation states the operation takes 20 milliseconds. You cannot necessarily conclude that the operation takes 20 milliseconds under every meaningful circumstance. For that, you would want to know something about the hardware, workload, test conditions, sample size, measurement method and what happens when the database contains rather more than twelve rows. Cybersecurity deserves the same discipline. If a company describes its product as providing continuous penetration testing, that description is useful evidence. It is evidence of what the vendor says the product does. It is not automatically independent proof of how comprehensively, reliably or effectively the product performs that function. This matters particularly when numbers appear. A detection rate of 99.8 per cent looks impressively authoritative. Numbers have that effect. Add a decimal place and almost anything begins to resemble physics. But before treating 99.8 per cent as an established property of the product, it is reasonable to ask who conducted the test, what was tested, against what, when, under which configuration, how large the sample was and whether the company selling the product paid for the research. None of those questions makes the statistic false. They tell us what sort of evidence it is. That distinction matters. Research is not hands-on testing There is a peculiar habit in online software publishing whereby the phrase "hands-on review" sometimes appears to mean that somebody looked at the company's website rather carefully. These are different activities. Reading technical documentation is research. Watching a product demonstration is observation. Installing the product and using it is experience. Designing controlled scenarios to establish how it behaves is testing. None of these is worthless, but neither should one masquerade as another. This is especially important in cybersecurity, because meaningful testing can be difficult. A reviewer can install a password manager and determine fairly quickly whether importing credentials works. Evaluating an enterprise penetration-testing platform is another matter. One needs an appropriate environment, defined targets, technical knowledge, controlled conditions, permission to conduct the activity and some idea of what the expected result ought to be. Clicking around a dashboard for twenty minutes and announcing that the product provides excellent protection would be wonderfully efficient. It would not tell us much. For that reason, TTML distinguishes documentation-based research from experience-informed research and genuine testing. If we have not tested something, we should not write as though we have. There is nothing wrong with saying, "According to the vendor's documentation..." Those few words tell the reader something important about the provenance of the claim. “Not confirmed” is a useful answer Databases dislike uncertainty. People building databases often dislike it even more. There is an empty field, and the empty field wants to be filled. Suppose we are comparing two security products and encounter the question, "Does Product B offer a free trial?" We cannot find one. The temptation is to write "No" and move on. The table is complete and everybody is happy. Unfortunately, we have answered a different question. What we established was that we could not confirm that Product B offers a free trial. We did not establish that Product B does not offer one. Those statements are not equivalent. This is why TTML uses phrases such as "Not confirmed", "Not publicly listed" and "Pending editorial review". They are not particularly glamorous, but uncertainty is itself information. When the evidence does not support either yes or no, the accurate answer is sometimes neither. Developers ought to be comfortable with this idea. There is, after all, a reason programming languages have spent so much time arguing about null. Compliance is not a feature you switch on Cybersecurity websites contain an impressive collection of capital letters: SOC 2, ISO 27001, PCI DSS, HIPAA, GDPR. Arrange enough of them in a row and an otherwise ordinary SaaS homepage begins to look as though security clearance is required to enter. The difficulty is that several quite different claims may sit behind a compliance logo. A vendor may hold a certification. Its software may provide controls useful to organisations pursuing certification. A platform may collect evidence useful during an audit. A service may support activities associated with a regulatory requirement. These things are related, but they are not identical. In particular, buying software does not ordinarily make an organisation compliant. Compliance usually involves technology, processes, controls, people, documentation and continued organisational behaviour. Installing a product and declaring the problem solved would be rather like buying a fire extinguisher and assuming that every fire regulation affecting the building has now been satisfied. So when reviewing security software, we try to preserve the actual meaning of the claim. "Supports compliance with" should not quietly turn into "makes you compliant with". "Designed to help satisfy controls" should not become "certified for". The difference may be only a few words, but they are important words. Comparisons make weak evidence worse A badly researched product review can misrepresent one product. A badly researched comparison can misrepresent two products and then manufacture a winner. This is quite an achievement. Comparison pages encourage certainty because their structure demands answers. Which product is cheaper? Which provides better protection? Which is easier to deploy? Which supports more compliance frameworks? Which should you buy? A table containing seventeen instances of "Not confirmed" is admittedly less satisfying than one full of green ticks and red crosses. But the purpose of the table is not to satisfy the table. It is to help somebody make a decision. Suppose Product A publicly documents ten capabilities and Product B publicly documents seven. It is tempting to conclude that Product A has more capabilities. Perhaps it does. What we actually know, however, is that we found more documented capabilities for Product A. Product B may lack the other three, or its documentation may be worse, or the same functions may exist under different names, or we may simply have failed to find them. That is why comparisons need evidence on both sides. Otherwise uncertainty has an unpleasant habit of turning into a competitive disadvantage, and the vendor with the better documentation wins rather than the vendor with the better product. A good product can still be the wrong recommendation Eventually, research has to become useful. Readers are not studying cyb