Bot Detection False Positives: How to Actually Test Accuracy

The fastest way to lose confidence in bot protection is not to miss a bot. It is to block a real customer. A missed scraper costs bandwidth or content. A blocked customer costs a sale, a support escalation, and trust that took months to earn. Which is why a bot detection benchmark cannot stop at a single accuracy number. It has to answer a harder question: what happens to real people when this policy leaves the dashboard and starts controlling traffic? A false positive is a business event In bot detection, a false positive is legitimate traffic classified as automated. The technical label matters; the consequence matters more. The same wrong classification produces very different outcomes: | Route | Possible false-positive cost | |---|---| | Public article | One page view is challenged or delayed | | Login | A customer cannot reach their account | | Password reset | A locked-out user cannot recover access | | Checkout | Revenue is interrupted at the point of purchase | | Public API | A partner integration begins failing | | Account creation | A legitimate prospect cannot start a trial | An aggregate false-positive rate hides these differences. A vendor can report a low global rate while causing concentrated damage on one browser, one mobile network, or one high-value route. So the useful question is not "what is the false-positive rate?" It is: how many known-human sessions did this rule challenge or block on each protected route, and what happened next? Accuracy is usually the wrong headline metric Bot traffic is a class-imbalanced problem. Most requests on a customer-facing app are legitimate; the attacks worth stopping are a small slice. In that setting a large accuracy percentage can describe a weak detector. Suppose a site receives 100,000 requests: - 99,000 are legitimate - 1,000 are automated attacks - The detector catches 900 attacks - It also flags 200 legitimate requests The detector is 99.7% accurate. That sounds excellent. It is also blocking or challenging 200 real requests, missing 100 attacks, and producing a bot verdict that is wrong almost one time in six. Use a small set of metrics together instead: Precision = true bot detections / all bot detections Recall = true bot detections / all actual bot attempts False-positive rate = false bot detections / all known-human requests In the example: precision 81.8%, recall 90%, false-positive rate ~0.2%. At one million legitimate requests per day, that last number is about 2,000 customer requests per day receiving the wrong treatment. Google's classification metrics guide makes the tradeoff clear: changing a threshold changes all three. There is no threshold you can evaluate independently of the cost of each kind of mistake. Which usually leads to three operating goals: - Hard blocks favor precision. The evidence should be strong enough that a human is very unlikely to match it. - Challenges balance precision and recall. They provide a recovery path for ambiguous traffic. - Monitoring favors recall. A broad signal is fine when a person or later rule reviews it before enforcement. One score should not control all three actions. Build a benchmark that resembles production A clean lab set with one Chrome version and a few obvious Selenium scripts proves the code runs. It does not establish that the detector is safe for customers. Build from three label groups. Known-human traffic Strong human labels come from successfully authenticated sessions, completed purchases that were not reversed, support-confirmed sessions, or employees following a controlled test plan. None is perfect alone - the point is traffic with independent evidence that a real person completed a meaningful action. Keep the sample representative across: - Desktop and mobile browsers - Older devices and slow connections - Corporate networks, universities, and carrier-grade NAT - VPNs and privacy tools your customers actually use - Assistive technology and keyboard-only navigation - Logged-in customers, anonymous visitors, and partner users - Every route where the policy may eventually enforce If the known-human set contains only employees on recent MacBooks, the benchmark is measuring employee laptops, not customers. Known automation Run controlled clients with Playwright, Puppeteer, Selenium, curl, and any stack relevant to your app. Include slow bots, distributed low-volume clients, headless browsers, and scripts carrying realistic headers. Trusted automation belongs in the set too. Search crawlers, uptime monitors, accessibility scanners, payment callbacks, and partner integrations are automated - that does not make them hostile. Unknown traffic Leave genuinely ambiguous traffic labeled unknown. Do not call every session that failed to convert a bot, and do not call every session that passed a JavaScript check human. Those shortcuts make the detector's own assumptions part of its ground truth. Freeze the policy during each test Record the exact configuration behind every verdict: engine version, rule and threshold version, signals that fired, score and proposed action, route group, session identifier, timestamp, allowlist decision. If thresholds change halfway through a test without a version marker, your final precision number combines two different systems. { "policy_version": "checkout-2026-09-01.1", "route_group": "checkout", "score": 72, "would_action": "challenge", "signals": ["headless_mismatch", "velocity_anomaly"], "identity": "unverified", "outcome": "purchase_completed" } The important field is outcome . Without it you can count detections but cannot tell whether a proposed action would have interrupted a customer. Start in shadow mode Shadow mode evaluates every request but does not change the response. A request that would have been challenged gets the normal page; a request that would have been blocked reaches the application. The proposed action and its evidence are logged. This is the safest place to tune thresholds, because the detector sees real traffic while mistakes stay observable rather than customer-facing. For every route group, answer: - How many sessions would have been allowed, challenged, or blocked? - How many would-block sessions later logged in, purchased, submitted a valid form, or called an authenticated API? - Which rules contribute most of the false-positive candidates? - Are errors concentrated by browser, device, geography, ASN, customer, or integration? - How much attack traffic would each threshold miss? Don't declare victory after a quiet afternoon. The sample should span weekday and weekend behavior, billing cycles, product launches, and campaigns. The rule of three The absence of an observed false positive is not proof the true rate is zero. A useful rough check: if a test observes zero errors in N independent known-human sessions, the upper edge of a rough 95% confidence interval is about 3/N . Zero errors in 1,000 sessions only supports a rate below roughly 0.3%. Zero errors in 100,000 sessions supports a much tighter claim. Report counts beside rates so a reader can tell whether "100% recall" means 2 of 2 attacks or 20,000 of 20,000. Test low-volume attacks without fooling yourself A credential stuffer sending two attempts per IP per day never creates an obvious spike. A scraper taking one page every few minutes blends into human traffic. If the positive class holds only a handful of confirmed attacks, one mislabeled session swings precision dramatically. - Extend the observation window. Collect enough normal traffic to see rare customer conditions, and enough attack traffic to be more than a one-day anecdote. - Replay known attacks. Recorded sequences let you compare policy versions against identical inputs. Keep replay results separate from live ones - a recording can't reproduce every timing and network condition. - Run controlled red-team traffic. Throttle your own automation to the rate a real attacker would use. Use realistic sessions and route order instead of hammering one endpoint. - Measure evidence, not just volume. A low request rate doesn't erase other evidence: failed identity verification, a composite fingerprint reused across accounts, a decoy link followed, an impossible field submitted, machine-like workflow consistency. Rate should be one signal, not the whole detector. Verify good bots before you allow them A crawler allowlist reduces false positives only if it verifies identity. Matching Googlebot in a User-Agent creates a bypass, because any client can send the same text - Google's own documentation warns the User-Agent is commonly spoofed and recommends verifying via published IP ranges or reverse DNS with forward confirmation. Declared identity -> verify source or signature -> verified: apply the crawler or partner policy -> failed: treat as impersonation evidence -> unavailable: keep unverified, avoid claiming certainty Don't turn a failed lookup into an automatic block if the source data may be stale. Record why verification failed and choose a route-appropriate fallback - public content can often fail open, sensitive APIs may require a service token. Shared IP addresses break simple enforcement An IP address is a network location, not a person. One address may represent an office, university, hotel, mobile carrier, VPN exit, or large customer integration. One abusive client behind that address does not make every other client hostile. This is exactly why an IP-only block looks accurate in a lab and fails in production: the test environment assigns one address per client, while production puts thousands of unrelated sessions behind one egress point. Treat IP reputation and request rate as context. Correlate with session evidence, authentication state, route, TLS and browser characteristics, and behavior. When uncertainty remains, challenge the session rather than blocking the address. Test rate limits from a shared-network simulator: send legitimate traffic from many independent sessions through one source address, t