Don't Rate-Limit Googlebot Like a Proxy Farm: Route Crawlers by IP Signal

Your content and API edge sees two kinds of automation that look similar in the logs: search-engine crawlers you usually want indexed, and scrapers that rotate through proxy exits to drain pages, prices, or inventory.

A single requests-per-IP throttle treats them the same. That is how you starve SEO while the farm simply dials a fresh residential address.

IP99 by ThreatHunter puts a typed signal on the free GET /v1/ip/{ip} response for the first case: crawler means search-engine crawler egress. Official docs treat it as benign - it does not participate in scoring and never becomes a scored piece of evidence.

Proxy farms show up under different slugs (proxy, vpn, dialup_pool, cloud_phone, …) with a score that measures freshness and strength of verifiable evidence, not "probability this visitor is an attacker."

What you are actually sorting

Official signal table (ten slugs): eight risk labels (proxy, vpn, dialup_pool, hijacked_proxy, osint, cloud_service, cloud_phone, plus crawler listed among them but benign), and two network attributes (hosting, mobile) that are not risk verdicts on their own.

Google Public DNS (8.8.8.8) carries hosting with score 0 - a reminder that "datacenter space" ≠ "scraper."

From IP99's own content/API anti-crawl note: scrapers rotate exits across proxy pools; routing by risk type (and share scope when you have it) mis-fires less than pure frequency limits.

Triage order at the edge

• evidence_state - can this conclusion be trusted now?
  • active = fresh evidence still scoring
  • stale = seen before, aged out, score 0 (weak feature only)
  • none = never seen (coverage gap), not "safe."
• risk.signals - why it matches. If crawler is present, treat as known search egress: do not fold it into your "proxy farm" ban path. If proxy-class signals fire without crawler, that is the scrape / abuse path.
• risk.score / risk.level - only after state. Official disposition guidance (generic, not a contract promise):
  • 0 → do not stamp "clean"
  • 1-29 → log/observe
  • 30-69 → verify (CAPTCHA, second factor, rate limit, demote) rather than hard-ban
  • 70-100 → block when business cost allows.
• Share scope - Pulse share_tier (T0-T3) if you have professional history; on free responses use usage_type (IDC vs DYN / MOB / GTW) as a rough prior before one ban equals one building.
• degraded: true - discount the answer when sources are partial.

UA strings alone are not enough: scrapers forge Googlebot. Combine UA with egress identity (published bot prefixes where vendors publish them, plus an IP risk lookup).

IP99's crawler tag is the supply-side observation that the address is search-crawler infrastructure - complementary to vendor prefix lists, not a replacement for verifying the ones you care about most.

Live curls

2026-10-29, no API key. Anonymous quota: 500 lookups/day per client IP, 60/min. Sign in for 5,000/day.

Overseas self-serve Pro / Pro+ buys volume, not a different score - official line: you pay for volume, not a better answer. Per-item evidence rows, share_tier, and ?at= history are a separate professional / contracted Pulse capability.

Datacenter control: hosting attribute, score 0 - not a scraper by itself:

curl https://ip99.com/v1/ip/8.8.8.8

Docs demo residential address - mask the last octet in tickets/logs:

curl https://ip99.com/v1/ip/27.22.66.200

Remaining anon quota:

curl https://ip99.com/v1/me

Sample A: hosting ≠ crawler ≠ proxy farm

Captured ~2026-10-29 13:08 UTC:

{
  "ip": "8.8.8.8",
  "computed_at": "2026-10-29T13:08:06Z",
  "evidence_state": "none",
  "geo": {
    "country": "US",
    "region": "California",
    "city": "Mountain View",
    "tz": "America/Los_Angeles"
  },
  "network": {
    "asn": 15169,
    "usage_type": "IDC"
  },
  "risk": {
    "score": 0,
    "level": "none",
    "signals": ["hosting"]
  }
}

usage_type: IDC plus hosting describes where the address lives. There is no crawler and no proxy-class signal. A rule that bans every IDC egress will also ban DNS resolvers, CI runners, and your monitoring probes.

Sample B: residential proxy churn

Same window; last octet shown because this is IP99's documented demo address - still mask real customer IPs in your tickets:

{
  "ip": "27.22.66.200",
  "computed_at": "2026-10-29T13:08:32Z",
  "evidence_state": "active",
  "geo": {
    "country": "CN",
    "region": "Hubei",
    "city": "Xiangyang",
    "district": "Yicheng",
    "tz": "Asia/Shanghai"
  },
  "network": {
    "asn": 4134,
    "usage_type": "DYN"
  },
  "risk": {
    "score": 8,
    "level": "low",
    "signals": ["dialup_pool", "hijacked_proxy", "proxy"]
  }
}

This is residential broadband with dial-up-pool / proxy evidence - the rotating-exit pattern scrapers buy. Disposition differs from Googlebot: prefer verify / tighter crawl budget / delay expensive endpoints. hijacked_proxy still argues for careful appeals (the line owner may be a victim), not "delete the building forever."

When your edge does see crawler (often on known search infrastructure), route it to the allow / polite-crawl lane even if the same IP sits in hosting space. Do not let a global "block hosting" or "block anything with a risk slug" rule catch it.

A minimal edge hook

Server-observed client IP only - never trust a body field:

IP="$CLIENT_IP"
RESP=$(curl -sS "https://ip99.com/v1/ip/${IP}")

Pseudocode:

# state = RESP.evidence_state
# signals = RESP.risk.signals or []
# if "crawler" in signals:
#   allow_or_polite_budget()
# elif state == "active" and any(s in signals for s in
#   ("proxy","vpn","dialup_pool","cloud_phone","cloud_service")):
#   if RESP.risk.score >= 70: challenge_or_block()
#   elif RESP.risk.score >= 30: captcha_or_slow()
#   else: observe()
# else:
#   normal_rate_limits()

Re-query at decision time. Do not cache a verdict across days: scores decay with evidence age, and evidence_state is how you tell "seen before, expired" from "never seen." Store computed_at with the decision if you may need to defend it later.

Context from this week's proxy exits

Between 22-28 October 2026 (UTC), the IP99 risk feed captured 11,196,585 proxy exit addresses it had not seen in the previous 7 days - about 80% residential broadband, datacenter only ~2.3%, from 68 upstream sources (proxy-exits).

Most of the scrape supply you are fighting is not sitting in the ASN blocklist you already maintain. That is why signal-aware routing beats "ban hosting + rate-limit everyone else."

Try it

Free lookup (no key): ip99.com or curl https://ip99.com/v1/ip/8.8.8.8. Field reference: API docs. Agents can call the same verdict as MCP tool ip_risk_lookup at https://ip99.com/mcp.