200 OK Is Not Enough: Why Bot-Protected Sites Still Return Bad Data

Your crawl job finished successfully. That doesn't mean it got the data.

Every scraping pipeline has a monitoring dashboard, and every monitoring dashboard has the same blind spot: it tracks whether requests succeeded, not whether the content that came back was real. A job that completes with a wall of green 200 status codes looks healthy. It can also be quietly wrong, page after page, for weeks, because a 200 response only tells you the server accepted the request. It says nothing about whether you're looking at the actual page or a version built specifically for visitors the site doesn't fully trust.

That gap between "the request succeeded" and "the data is correct" is where most silent pipeline failures live, and it's getting wider as anti-bot systems get more sophisticated about what they serve instead of an outright block.

What a "successful" response can actually contain

A block used to be simple to detect: a 403, a 429, a connection reset. Modern anti-bot systems increasingly prefer a different approach, because an obvious block tells the requester exactly what happened and invites a fix. A soft block, served with a 200, doesn't.

In practice, that 200 can be:

• A challenge page: an interstitial that looks like real content in the raw response but is actually a JavaScript-driven verification step (a "just a moment" style page, a hidden CAPTCHA iframe, a redirect loop disguised as a normal page load).
• A cached fragment: an old snapshot of the page served to anything that looks automated, so the price, availability, or listing you scraped is stale even though the request itself worked fine.
• An empty state: a search results page or listing that legitimately returns "no results" to a request pattern the site doesn't recognize, even though a real visitor would see dozens of items.
• A partial HTML shell: the server response contains the page skeleton, but the actual content only renders after JavaScript executes in a real browser, so a plain HTTP client gets a technically valid, functionally empty document.

None of these trigger an error. All of them will pass a naive health check that only looks at the status code.

Here's a concrete version of how this plays out.

An e-commerce price monitoring job hits a retailer's product pages every four hours. One day, without any code change on either side, a subset of requests starts getting a cached response: the retailer's CDN serves a snapshot from a few hours earlier to traffic patterns it doesn't fully trust, rather than routing to the live pricing service. The crawl finishes in normal time. Every response is 200. The HTML looks completely legitimate, because it is legitimate, it's just not current. The price feed downstream keeps updating on schedule with numbers that are quietly a few hours to a few days stale, and nothing in the pipeline's own metrics shows anything unusual, because nothing about the request-response cycle failed.

Why status-code monitoring misses all of it

Most scraping pipelines validate success the same way:

• Did the request return 200?
• Did the response have a non-trivial size?
• Did the job finish without an exception?

That's a reasonable first filter, but it's checking whether the pipe worked, not whether what flowed through it was real. A soft block or a cached fragment will often pass every one of those checks. The response is 200. It has content, sometimes a lot of it. The job finishes cleanly. The only way to catch the problem is to look at what the content actually says, not just whether it arrived.

Validating content, not just responses

The fix isn't a single trick, it's a layer of validation that runs after every fetch, specific to what a real version of that page should contain. A few patterns cover most of the failure modes above:

def validate_response(html, expected_selectors, baseline_length, known_challenge_markers):
    # 1. Check for known soft-block / challenge page signatures
    lowered = html.lower()
    for marker in known_challenge_markers:
        if marker in lowered:
            return "soft_block", marker

    # 2. Check the page actually contains what it should
    missing = [sel for sel in expected_selectors if sel not in html]
    if missing:
        return "content_missing", missing

    # 3. Check size against a rolling baseline for this source
    if len(html) < baseline_length * 0.5:
        return "suspiciously_short", len(html)

    return "ok", None

This is deliberately simple, and that's the point: even lightweight checks like these catch a large share of soft blocks and empty states before they reach a dashboard.

A few refinements make it considerably stronger:

• Track a rolling baseline response size and structure per source rather than a fixed number, since normal pages vary.
• Maintain a small, source-specific list of challenge markers (specific strings, class names, or redirect patterns each anti-bot vendor's interstitial tends to use) and update it when a source's defenses change.
• For sources that require JavaScript execution to render the real content, don't rely on the raw HTTP response at all; render the page in a real browser context and validate against the rendered DOM instead of the initial payload.

That last case is common enough on modern sites that it deserves its own approach, and it's worth going deeper on crawling techniques for JavaScript-heavy websites if that's where most of your sources live.

Treat this as ongoing, not a one-time fix

The uncomfortable part of all this is that validation rules go stale the same way scraping rules do. A challenge marker that reliably caught a soft block six months ago might change wording tomorrow. A baseline response size that was accurate in January can drift as a source's page templates change.

Content validation isn't a script you write once and forget; it's a monitoring layer that needs the same ongoing attention as the crawling logic itself, because the sites on the other end aren't standing still either.

The takeaway

If your pipeline's definition of success stops at the HTTP status code, you almost certainly have data quality problems you haven't found yet, not because your crawler is broken, but because "it ran without errors" and "it collected the real page" have quietly become two different claims. Building even basic content validation into the pipeline, and treating it as something that needs maintenance just like the crawling logic itself, is what closes that gap before a stale price or an empty result set makes it into a report someone downstream trusts.