Your robots.txt decides whether ChatGPT can cite you. Here's the 5-minute check.

More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue link. Whether those answers can ever cite you is decided by one file: robots.txt . And most robots.txt files decide it by accident. Two kinds of AI crawlers - and only one of them matters for citations | Kind | User agents | Blocking it means | |---|---|---| | Training | GPTBot , ClaudeBot , Google-Extended , Applebot-Extended , CCBot | your content isn't used to train models | | Retrieval | OAI-SearchBot , ChatGPT-User , Claude-SearchBot , Claude-User , PerplexityBot | you can never be cited in an AI answer - no link, no attribution, ever | Blocking training crawlers is a legitimate choice. Blocking retrieval agents is almost never what a site intended - it just silently removes you from AI answers. The blanket-block mistake The "block the AI bots" wave of 2024/25 produced a lot of this: User-agent: GPTBot User-agent: ChatGPT-User User-agent: ClaudeBot Disallow: / That single group blocks training and retrieval - worst of both worlds if you still hold the training position but wanted visibility. I checked major news sites this week: nytimes.com allows 1 of 7 retrieval agents, reuters.com, theverge.com and wired.com 2 of 7 each. None of them is fully citable - and these are sites whose entire business is being the cited source. A deliberate policy - citable, but not training material - looks like this: User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: / The CDN plot twist: rules you never wrote Here's how I fell into this myself. My own site's served robots.txt contained a "managed" block list my CDN (Cloudflare) had injected above my hand-written rules - blocking the same crawlers my own section explicitly allowed, plus a Content-Signal: line that is an express reservation of rights under Article 4 of the EU DSM Directive. A legal statement, published in my name, that I had never written. Two lessons: - Read the served file, not your source file. curl https://yoursite/robots.txt and look for sections you didn't write. - Contradictory groups for the same crawler are effectively undefined behavior. The spec's tie-breaking (longest match wins, Allow beatsDisallow on equal length, a group naming the agent beats* ) resolves conflicts within a group - but when two groups target the same bot with opposite rules, what happens depends on each vendor's implementation. You genuinely don't know what you're telling that crawler. Beyond robots.txt: two more silent killers - Pages that are empty without JavaScript. Retrieval agents don't render. A page Google ranks fine (Google renders) can be blank to every AI system. Test: curl your page and check whether the text is in the HTML. - Missing attribution signals. AI answers lift and attribute passages more readily from pages with an author, a date, question-shaped headings and FAQPage markup. The 5-minute check curl https://yoursite/robots.txt - Search it for the five retrieval agents above. Any Disallow that hits them = you're out of AI answers. - Look for sections and Content-Signal: lines you didn't write. - curl a key page and check the text is present without JavaScript. For one site, that's honestly enough. I also built the bulk/scheduled version - a technical SEO audit that includes exactly this citability check per vendor, plus a diff against the previous run: SEO Audit + AI Visibility on Apify. Disclosure: that's my tool; the manual check above needs nothing but curl. Top comments (0)