robots.txt for AI crawlers: let search bots in, keep training bots out

You can block the bots that train AI models without disappearing from ChatGPT, Perplexity or Claude answers. The trick is to treat AI bots as two different groups and to name each bot explicitly in robots.txt.

Two kinds of AI bots

Search and assistant bots fetch your pages to answer a user's question, and they are the ones that bring you citations:

• OAI-SearchBot and ChatGPT-User (OpenAI)
• PerplexityBot and Perplexity-User (Perplexity)
• Claude-SearchBot and Claude-User (Anthropic)

Training-only bots collect pages to train models:

• GPTBot (OpenAI)
• ClaudeBot (Anthropic)
• CCBot (Common Crawl)

Blocking the second group is a legitimate choice, and it doesn't cost you citations. Blocking the first group does.

The rules

# Search and assistant bots: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Training-only bots: your call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

Why name every bot

Under RFC 9309, a crawler follows the group that names it, and the * group applies only if no group names it. So an explicit Allow for each search bot survives even if someone later adds a catch-all User-agent: * / Disallow: /.

Google-Extended is a separate call: blocking it also affects Gemini's answers.

robots.txt is only a request

A CDN or firewall can stop bots before they ever read robots.txt. Since July 2025, Cloudflare blocks known AI crawlers by default on new domains and asks new sign-ups whether to let them through (MIT Technology Review, 1 July 2025).

Check your Cloudflare setting, your host's anti-bot protection, any WAF and your security plugins.

Check who really got through

Search your server logs for the bot names. But the name in a request can be faked, so verify the IP: OpenAI publishes its bots' IP addresses, and Google's bots are verified with a reverse DNS lookup.

Your 15-minute checklist

• Rewrite robots.txt with the two groups above.
• Check CDN and hosting bot settings.
• Find at least one verified visit from a search bot in your logs.

Full guide (WordPress included): https://citeproof.co/blog/ai-crawlers-robots-txt-llms-txt-wordpress?utm_source=devto&utm_medium=parasite&utm_campaign=block-gptbot

I'm the founder of CiteProof, which measures whether AI assistants cite a site. The rules above work with or without any tool.