DEV Community

Why HTTP 200 doesn't mean the account exists: false positives across 3,450 sites

If you have ever written a username checker, you have written this function:

def exists(session, url):
    r = session.get(url, timeout=10)
    return r.status_code == 200

It works on about nine sites in ten. The other tenth is where username enumeration quietly goes wrong, and across 3,457 sites that tenth is several hundred sites returning answers you cannot trust.

We ran 60 handles against 3,166 sites - 189,960 checks - then opened the suspicious results by hand. Here are the failure modes and how we classify them now.

The trick: a handle that cannot exist, found anyway

Fifteen of our test handles were 13 to 18 random characters, things like hollowpebble1650. Nobody has registered those anywhere. They are a control group, and you need no ground truth to use them: any site reporting "found" for one is wrong by construction.

Five sites reported found for all fifteen. Opening those by hand:

  • Three served a bot challenge with status 200. A self-hosted code forge, a link aggregator and a video site each returned a "making sure you're not a bot" interstitial. Status 200, HTML body, no profile. Any status-code check counts it as a hit.
  • One forum returns its app shell for every path. A single-page app that resolves client-side: the server answers 200 for /u/anything and the 404 only happens in JavaScript you never execute.
  • One music community disagreed with itself. By hand we got "not found". From our server we got a success code. Same URL, different answer - geography, TLS fingerprint or datacentre IP reputation, we never pinned it down.

That last one is the uncomfortable case, because no amount of response parsing fixes it. The server is genuinely showing you a different page.

Why status == 200 is the wrong question

The question you want answered is "does a profile exist here", and HTTP gives you no field for that. What you get instead:

What happened What you often see
Profile exists 200 + profile markup
Profile does not exist 404, or 200 + an empty-state page, or a 302 to the homepage
You are rate-limited 429, or 200 + a challenge page, or a 503
Bot detection fired 403, or 200 + an interstitial
Site is down 502, or 200 + a status page

Four of those five produce a 200 somewhere. A boolean return type cannot represent this, which is the actual bug: not the comparison, the signature.

Four states, not two

We return one of four values:

class Result(Enum):
    FOUND = "found"
    NOT_FOUND = "not_found"
    BLOCKED = "blocked"  # challenged, rate-limited, refused
    UNKNOWN = "unknown"  # answered, but unreadably

Across our 189,960 checks: 81.0% not found, 8.8% blocked, 5.1% unknown or error, 4.9% found. So about one check in seven lands in blocked or unknown. If your tool reports only taken-or-available, those 14% have to go somewhere, and if they land in "available" you are wrong one time in seven. That is not a rounding error in a tool whose entire output is a boolean.

Blocking is also wildly uneven by category, which means a single global timeout and retry policy is the wrong shape:

Category Checks blocked
Shopping (38 sites) 34.2%
Hobby (84) 23.6%
Gaming (201) 20.2%
Misc (295) 19.2%
Coding (294) 6.2%
Social (1,608) 4.8%
Video 0.1%

Detecting the catch-all site automatically

The manual version does not scale to 3,457 sites, so the control-group trick becomes a test. Generate handles no human would register, probe each site with them, and flag any site that claims them:

CATCH_ALL_THRESHOLD = 0.8

def is_catch_all(site, probes):
    """probes: long random handles that cannot exist anywhere."""
    hits = sum(1 for h in probes if check(site, h) is Result.FOUND)
    return hits / len(probes) >= CATCH_ALL_THRESHOLD

We treat a site as catch-all at 80% or above and exclude it from counts. Five of 3,166 met it. That sounds small until you notice what it did to our headline number: random handles averaged 11.9 "found" sites before the filter and 6.9 after. Five sites out of three thousand were inflating the average by 42%.

Run this on a schedule, not once. Sites add Cloudflare, migrate to SPAs and change their 404 behaviour without telling you.

Detection rules that do not rely on the status code

Per-site rules, in rough order of how much we trust them:

  • A string that only a real profile page contains - not the handle itself, which appears on plenty of "user not found" pages. Something structural: a follower count element, a profile schema block.
  • A string that only the empty state contains, matched as a negative. Fragile: these get reworded.
  • Redirect behaviour. Many sites 302 unknown handles to the homepage or a search page; the final URL tells you more than the status.
  • Response length bands. Crude, but a challenge page and a profile page differ by an order of magnitude, and it catches regressions in the rules above.

And two signals that flip a check to BLOCKED or UNKNOWN regardless of what the content rule says: a known challenge-page fingerprint, and a response that is 200 but implausibly short.

What this means if you are building one

  • Return an enum, not a bool. Everything else follows from the signature.
  • Keep a control group of impossible handles and run it continuously.
  • Surface blocked and unknown in the UI. Users would rather check nine sites by hand than be wrong about one.
  • Re-test your detection rules on a schedule. The web rots underneath them.
  • Publish your error rate. Any tool claiming 100% reliable results across thousands of sites has not measured it.

Full methodology, limitations and the raw CSVs are published under CC BY 4.0: the 60-handle study across 3,166 sites, and the research index with a nightly site-health tracker and a false-positive benchmark of Sherlock, Maigret and Blackbird on identical handles.

Disclosure: I work on FreeUsername, the checker these numbers came from.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.