GPTBot in robots.txt: the hosting toggle developers need to check

Your robots.txt may express an AI policy you did not write. We checked the homepage and robots.txt of 9,037 live AI tools listed on directree on 6 and 7 September 2026. Of those, 945 explicitly disallow OpenAI’s GPTBot in its own user-agent group: 10.5% of the sample.

Treat AI crawler rules as deployment configuration. Review them when you change hosting, enable a CDN feature, adopt a starter template, or hand site operations to someone else. Read the full research and methodology.

GPTBot, Search, and User Browsing Are Separate

A common configuration blocks model training while keeping a site available in AI-assisted search and browsing:

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /

These are separate crawlers with separate purposes. In our sample, 839 of the 945 sites that block GPTBot, or 88.8%, still allow OAI-SearchBot. That is a deliberate and useful distinction if your goal is to opt out of training while remaining eligible to be