133 of 10,099 Shopify stores block an AI crawler. Six block the one ChatGPT shops with.

Method

Merchants are told two opposite things about AI crawlers: block them, because they take your content and give nothing back; and admit them, because that is how a store gets into an AI shopping answer. Both assume a decision is being made. We wanted to know how many stores have made one, and which way.

Every store in a corpus of 10,099 known Shopify storefronts has its /robots.txt read as part of a scan. The file is parsed the way the major crawlers document parsing it: most specific User-agent group wins, * is the fallback, longest matching path rule wins, Allow beats Disallow on a tie. Each of twelve crawler names is asked one question: may it fetch /products/? A store counts as blocking a crawler when the answer is no.

A second reading looks at the product page for a meta name="robots" tag carrying noai or noimageai. Readings were taken between 29 August and 2 September 2026. Every store that blocked at least one crawler, or carried the tag, is one row in the CSV at the end. The rest of the corpus blocked nothing and is the denominator.

What this cannot see

A robots.txt is a request. A store can also block a crawler at the edge, with a bot-management rule or a firewall, and that block is invisible here because the scanner is not the crawler being blocked. Every count below is a floor.

Result

133 of 10,099 stores block at least one crawler: 1.32%.

"Fetches at answer time" marks the four crawlers that read a page, or index for a search result, at the moment a person is asking. The other eight crawl ahead of time, to train or ground a model. The split follows each operator's published description of the name.

The twelve are the names merchants' block lists actually carry, not every answer-time agent that exists; Anthropic's and Google's live-fetch agents are not among them.

How They Block

• 130 of 133 block with Disallow: /. The whole site, not the product pages.
• 3 block /products/ alone.
• 54 block exactly one crawler.
• 28 block the same eight, verbatim: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, meta-externalagent. 4 more block those eight and others. Identical lists do not arise independently; this one reads like a copied snippet, and it contains none of the four answer-time crawlers.
• 1 store blocks all twelve.

Training or Answering

Of the 133 stores that block anything, 120 block only training and grounding crawlers. 13 block at least one crawler that fetches at answer time; 1 of those blocks only answer-time crawlers, 12 block both kinds.

The clearest pair is OpenAI's. GPTBot trains; OAI-SearchBot is what ChatGPT's search and shopping results are built from. 77 stores block the first. 6 block the second, and every one of those 6 also blocks the first. The reverse, keep the shopping crawler out and let the training crawler in, happens on 0 stores.

No store in the corpus has decided to stay out of AI shopping answers. The stores that are out of them by robots.txt are there because a copied training opt-out happened to include the name.

The noai Tag

8 stores carry a noai or noimageai meta robots tag on their product page. 6 of the 8 block nothing at all in robots.txt. By their names, six of the eight are musicians' merchandise stores, which again looks like one template rather than eight decisions.

The tag is a separate mechanism and this study makes no claim about which crawlers honour it.

What It Does and Does Not Mean

Blocking AI crawlers is rare on Shopify. 98.7% of stores block none of the twelve by robots.txt. Where it happens it is mostly a whole-site rule aimed at training crawlers, and mostly one list. The shopping crawlers are almost never blocked.

13 stores in 10,099 block any of the four answer-time crawlers we checked. For the rest, whatever stands between the store and an AI shopping answer, it is not robots.txt. It is not a recommendation.

A merchant who does not want their catalogue in a training set has a reason, and a copied list that does the job is a fine way to act on it. The finding is narrower: the list most of them copied was written before the answer-time crawlers existed, so a store using it has made a decision about training and, without noticing, none about shopping.

It is a floor. Edge blocks are real and this cannot count them.

Reproduce It

curl -s https://EXAMPLE.com/robots.txt

Find the User-agent: group that names the crawler; if none does, the * group applies. Under that group, the longest rule matching /products/ decides, and Disallow: / matches everything.

All 139 rows, one per store, are in ai-crawler-blocking.csv: host, grade, date read, one column per crawler, whether the noai tag was present, and whether the rules were whole-site or product pages only. Every row names a domain whose robots.txt you can read yourself.

Originally published at shelfglance.com/research/ai-crawler-blocking, where the corpus of 10,099 scanned storefronts lives.