Your instinct is right. Pay Per Crawl is a clean piece of infrastructure - HTTP 402, signed crawler identity, Cloudflare as merchant of record - but it prices the wrong unit. A crawl is an input cost. The thing publishers actually lose is output: human visits, ad impressions, subscriptions, and the relationship with a reader. Those two numbers do not live on the same scale.

What Pay Per Crawl actually does

A site owner on Cloudflare can set one of three policies per known AI crawler: allow for free, block, or charge a single zone-wide price. A complying crawler either sends crawler-max-price / crawler-exact-price and gets 200 plus a crawler-charged header, or it gets 402 Payment Required with crawler-price. Cloudflare bills the crawler and sits in the middle. Re-crawls of the same URL are billed again. /robots.txt, sitemaps and a few well-known files stay free. WAF/bot blocks still override “charge.”

That is real progress versus the previous default of “we take everything and you get nothing.” It creates scarcity, which is the precondition for a market. It also forces crawlers to identify themselves instead of blending into generic bot noise.

It does not, however, restore the old search bargain: you may index me because you send people back.

Why a paid crawl is not a substitute for a visit

Search-era crawling was a loss leader that paid off in referrals. AI-era crawling is often extraction. Cloudflare’s own crawl-to-referral ratios made that explicit:

• Anthropic was measured around 38,000 crawls per referred visitor (and much worse earlier in 2025).
• OpenAI around 1,091 : 1.
• Perplexity around 194 : 1.

Even if you charge, say, $0.01 per successful fetch, the economics still fail on two sides at once.

Publisher side. One human visit on a media site is not $0.01. It is pageviews × RPM, plus newsletter signups, plus affiliate, plus the chance someone becomes a subscriber. A single AI answer that replaces that visit can wipe out dollars of expected value while the crawl invoice is cents. Pew found that when Google showed an AI summary, users clicked a traditional result about 8% of the time - roughly half the rate without a summary - and clicked a link inside the summary only about 1% of the time. Later click-through studies from Ahrefs and others put organic CTR drops on AI Overview queries in the 35-60% range. Some verticals on Cloudflare’s network saw human traffic fall 35-40% in under a year.

Crawler side. More than half of “good bot” crawls are re-fetches of pages that have not changed. Training already dominates crawler purpose (Cloudflare put training at ~52% of identified crawler requests by mid-2026, with mixed-use another ~36%). You can charge for every one of those wasted hits and still not be pricing use. A page crawled once can be baked into a model or cited in thousands of answers. A page crawled a thousand times might never appear in a single response. Pay-per-fetch cannot tell those stories apart.

So your sentence is the whole critique: the lost visit is the product. The crawl is only the warehouse forklift.

Cloudflare already admitted the unit is wrong

That is why they started talking about Pay Per Use on top of Pay Per Crawl: experiments with Ceramic.ai (pay when content appears in a result) and You.com (pay when an agent actually needs a specific premium page). Payment follows citation or consumption, not the fetch. They also started shipping AEO-style reporting - which queries surfaced you, which snippet, which rank - because “we crawled you” is not a business metric.

That is closer to justice than 402 alone. It is also much harder. Crawling is observable at the edge. Use lives inside someone else’s model, ranking stack, or chat transcript. You only get paid for use if the AI company reports it honestly, or if a third party can audit citations. The companies with the most leverage are the least eager to open that ledger.

The structural holes that Pay Per Crawl cannot close

1. Google is still the exception that breaks the market. Publishers cannot treat Googlebot like GPTBot. Search inclusion, Discover, and now AI Overviews/AI Mode ride the same crawler. If you charge or block Googlebot you risk disappearing from the remaining human funnel. Mixed-use crawlers are the whole point of Cloudflare’s later default-block push on ad-bearing pages (announced for mid-September 2026 on new/free-tier defaults). Until search and training/agent use are separate, verifiable crawlers, “charge AI” is optional for everyone except Google.

2. Identified crawlers are not the whole problem. Pay Per Crawl only works against parties that register keys, sign requests, and accept a Cloudflare invoice. Anonymous scrapers, residential-proxy farms, and “research” wrappers that never declare themselves just keep hitting 403 or spoofing browsers. The honest payers subsidize the dishonest ones unless blocking is aggressive and detection is good.

3. One price per zone is a blunt instrument. You cannot charge GPTBot $0.10 and a niche research crawler $0.001. You can only allow / charge / block per crawler, with a single price for everyone on “charge.” A homepage, a 3,000-word investigation, and a product listing are not the same good.

4. Charging does not bring the reader back. Even a well-priced crawl is a licensing fee for machines. It does not recreate brand, community, comments, or the next click on your site. If the user is satisfied in the chat window, the publisher becomes a silent wholesaler. That may be a viable business - wire services already live that way - but it is a different business than “we run a website.”

5. Blocking has its own tax. There is research suggesting publishers who blocked major LLM crawlers via robots.txt lost on the order of ~7% weekly human traffic within weeks - not because bots vanished, but because some discovery paths still run through those systems. Pure block vs pure charge is not free.

What “comparable” would actually require

If the goal is to make AI access as valuable as the visits it displaces, you need a stack, not a header:

• Separate crawlers by purpose - search index, training corpus, live RAG/agent fetch - with cryptographic identity, not a shared user-agent.
• Pay for use, not for GET. Citation, snippet display, answer grounding, or successful tool-use of a page. Crawl fees at most cover bandwidth and cache invalidation.
• Auditability. Publishers need query + URL + snippet logs, not a monthly Cloudflare payout line that says “12,441 charged requests.”
• Default deny for training on monetized pages, which is the direction Cloudflare is pushing, because without scarcity there is no price.
• A path that is not Cloudflare-only. HTTP 402 plus signatures can become a web standard. If it stays a CDN product, the open web outside that network remains a free quarry.

Pay Per Crawl is the right protocol experiment. Your objection is the right economic one. A paid forklift trip to the warehouse will never equal the customers who stopped walking into the shop because someone already read the label aloud in the street. The interesting question now is whether Pay Per Use can be measured tightly enough that “appeared in an answer” starts to look like a visit - or whether publishers simply have to accept they are becoming licensors, and price the license like one.