My Scanner Missed 93% of the Bugs - and That Was the Right First Result

The first time I ran my vulnerability scanner against the industry-standard benchmark, the bottom line of the scorer's report was this: $ python scripts/score_benchmark.py --findings out/java.findings.json \ --truth benchmark-java/expectedresults-1.2.csv OVERALL precision 0.60 recall 0.07 F1 0.13 # abridged Three numbers, and here is what each one means. Precision 0.60 - of all the alarms the scanner raised, 60% pointed at real bugs: when it spoke, it was right more often than not. Recall 0.07 - of all the real bugs in the benchmark, it found 7%. In the four vulnerability classes my scanner covers, the benchmark contains 777 real, labeled vulnerabilities; it missed 93% of the bugs it exists to find. F1 0.13 - precision and recall combined into one score (their harmonic mean), dragged down to almost nothing by that recall. My first instinct was to fix it before anyone saw it. Instead I saved the output, wrote the number into my benchmark log, and kept it - because that number was always going to be published, and this is the article that publishes it. The Context For the past months I've been deep in AI - reading, building, measuring. One of the projects that came out of it is an AI vulnerability scanner. The design in one sentence: deterministic static-analysis rules do all the searching, and an LLM judges each finding - is this a real bug or a false alarm? The full architecture gets its own article. This one is about the first measured number. The test set is the OWASP Benchmark - 2,740 labeled Java test cases, the standard exam for Java security scanners. In my scanner's four vulnerability classes (SQL injection, command injection, path traversal, XSS) there are 1,478 cases: 777 real vulnerabilities and 701 cases deliberately designed to bait scanners into raising false alarms. Every tool I compare against - Semgrep, CodeQL - takes the same exam, scored by the same scoring code. Same rules for everyone. New to this? Three words carry this article. A source is where untrusted input enters a program (request.getParameter("id") - anything an attacker can type). A sink is where that input becomes dangerous (executeUpdate(sql) - running it as a database command). A vulnerability is data flowing from a source to a sink without being cleaned on the way; that flowing data is called tainted, and tracking it is taint analysis - the scanner's whole job is finding those flows. Already know all that? Skip this box. Why 0.07 Was the Right First Result Here's what I had actually run: a spike. A deliberately minimal first version - one source pattern, getParameter , wired to a handful of sinks, pushed end-to-end through the whole pipeline: parse 2,740 test cases into a code graph, run the taint queries, emit findings, score them against the answer key. The spike's job was never to score well. Its job was to answer one simple question: does the machinery work at all? And the ugly number, read carefully, answered it: - Precision 0.60 - when the scanner did raise an alarm, it was usually right. The taint engine was tracking real flows correctly. - Recall 0.07 - it was blind to 93% of the bugs. The engine wasn't broken; its vocabulary was tiny. I was listening at one door of a building with many doors. That's not a broken idea. That's a correct, simple diagnosis of a too-narrow source list - delivered before I had invested weeks in the wrong layer. If the first number had been precision 0.10, I'd have had an engine problem, which is a rebuild. A recall problem is a list problem. Lists are fixable. Why the Obvious Moves Are Both Wrong Obvious move #1: don't tell anyone. Fix it quietly, publish only the final number, look competent. Almost everyone building in public does a version of this - the "overnight" success graph that starts at the first good result. The problem: every result in this series is a number from my own benchmark runs. There is no referee here - no third party checks my work before it ships; it is me, a scorer script, and you. A reader has exactly one way to judge numbers like that: the author's track record with results that hurt him. If every number I show you is a win, you have no reason to trust any of them. So the bad numbers ship too - and they ship first. I'm going to publish a head-to-head against Semgrep and CodeQL later, and when I claim a result there, I want the reader thinking "this is the person who published his own 0.07." Honesty is not a virtue here; it's infrastructure. Obvious move #2: carpet-bomb the rules. Recall too low? Add patterns! Match more names, loosen the regexes, taint everything - recall will climb. It will also destroy precision, and worse: after twenty simultaneous changes you cannot say which one did what. You've traded a measured system for a vibes system. What I did instead was slower and duller: read the benchmark's actual code, find what it really calls, add sources in order of how often the code uses them, and re-measure after every change. One variable at a time, one number per change. The Climb Fix 1: Learn the benchmark's vocabulary - recall 0.07 → 0.83 I surveyed which input methods the benchmark's code actually uses, counting files: getRequestURI in 724 files, getCookies in 664, getParameter in 538, getParameterValues in 510, getHeaders in 400, and on down the HTTP request surface. My spike had covered exactly one entry in that list. So sources became one shared definition - a single regex over fully-qualified method names, so each getter is bound to the type that makes it attacker-controlled. Condensed here; the full rule table gets its own article: // Queries run on Joern, an open-source code-analysis engine (Scala DSL). ".(HttpServletRequest|ServletRequest)\.(getParameter|getParameterNames|getHeader|" + "getHeaders|getCookies|getQueryString|getRequestURI|getInputStream|…)\b." + "|.Cookie\.(getValue|getName)\b." + "|.SeparateClassRequest\.(getTheParameter|getTheValue)\b." // the benchmark's request wrapper (.Cookie.getValue. matches only the cookie's getter - a bare match on getValue would match every getValue in existence.) Two more problems were hiding inside this step, and they were different problems: - 117 of the benchmark's real SQL injection cases never touch java.sql - they go through Spring'sJdbcTemplate instead. One new sink row took SQLi recall from 0.57 to 0.86. - 6,060 XSS sink calls were invisible for a reason outside my rules. Without the servlet library on the analysis path, the engine cannot work out what type response.getWriter() returns, so those calls could never match a type-based.Writer. pattern. The fix: also accept a sink when the receiver text - theresponse.getWriter() part as literally written in the code - matchesgetWriter|getOutputStream . That single change took XSS recall from 0.03 to 0.73. New score: precision 0.53, recall 0.83. Fix 2: 97 of the 130 remaining misses shared one missing source - 0.83 → 0.95 There were still 130 real bugs missing. I diffed the misses against the benchmark code, and 97 of them - three quarters of everything left - took their taint from a single method I hadn't listed: getParameterNames() . It's easy to see why it gets skipped. getParameter("id") returns a value the user typed - obviously dangerous. getParameterNames() returns the parameter names - and names feel like structure, not data. But the client chooses the names too. ? alert(1) =x is a query string any client can send, and then the name is exactly as attacker-controlled as the value. One name in a regex. Adding it recovered 93 real vulnerabilities on the spot: recall 0.83 → 0.95. (The other 4 of the 97 were also blocked by a second, separate problem - they return in Fix 3.) And here's the part that still bothers me: nothing ever crashed, warned, or looked wrong. A missing source fails silently. Without a labeled benchmark I would never have known. Fix 3: The taint bridge - 0.95 → 1.00 That left 37 misses - the 33 that never used getParameterNames , plus the 4 from Fix 2 that had a second problem - and every single one contained .split(...) . Instead of guessing, I measured the taint chain link by link on one failing case: the variable being split was reachable from the source. The split call itself - reachable. The array-index access on its result, param.split(" ")[0] - not reachable. Taint flowed correctly through split and died at the index operation. The engine ships a default rule for that operator, and overriding it changed nothing - the gap was in how the engine applies rules to that operator, not in anything I could configure away. So I built a bridge, with one condition that keeps it honest: an index access over a split -style call is promoted to an additional source only when the array it indexes is itself reachable from a real source. Indexing an untainted array stays untainted - that condition is the difference between a targeted fix and blanket over-tainting. All 37 misses recovered, at a cost of exactly three new false positives and about 84 seconds of extra work per scan - a fix that raised recall and precision at the same time. The Numbers The whole climb, one measured change at a time: | stage | precision | recall | F1 | |---|---|---|---| spike (getParameter only) | 0.60 | 0.07 | 0.13 | | + broadened sources, receiver-text sinks | 0.53 | 0.83 | 0.65 | + getParameterNames | 0.55 | 0.95 | 0.70 | | + index-access taint bridge | 0.56 | 1.00 | 0.72 | Zero false negatives. All four classes land at recall 1.00 - SQL injection 272 of 272 real bugs found, command injection 126 of 126, path traversal 133 of 133, XSS 246 of 246 - the same recall CodeQL achieves on the same 1,478 cases, scored by the same code. From a seven-row rule table, with no build step. Now the uncomfortable part, because this is an engineering log and not a launch post. Precision 0.56 means 614 false alarms, and this layer falls into 88% of the benchmark's designed traps - more than either incumbent (the head-to-head article prints the full comparison)