Cloudflare’s September 15 AI Crawler Update: What Actually Shipped

On September 15, 2026, Cloudflare shipped the control its earlier announcement did not describe: Disallow AI Training. The setting uses Bot Preference Sync to publish the applicable no-training preference in robots.txt, lets accountable mixed-use crawlers continue Search, and blocks other training crawlers. Cloudflare also made Block and Block on pages with ads apply to mixed-use crawlers, so those choices can now remove Applebot, Bingbot, and Googlebot from Search as well as Training. ⁹

The launch did not impose one blanket default. New domains are offered one of two editable presets: non-ad sites get Allow for Search, Training, and Agent; ad-monetized sites get Search: Allow, Training: Disallow AI Training, and Agent: Block on pages with ads. Existing domains were migrated from legacy settings to preserve their practical effect, rather than left untouched. ⁹

What are the three bot categories, and what does blocking each one cost?

The September change rides on a taxonomy Cloudflare introduced on July 1: every AI crawler request is classified as Search, Agent, or Training, and each category can be allowed or blocked independently. ¹ The names matter less than the trade each one represents. Three categories, three separate bets, one checkbox most people have never opened.

The settings are not interchangeable. Disallow AI Training is a Training-only preference designed to preserve Search access for accountable mixed-use crawlers. Block is enforcement: it now includes mixed-use crawlers, so using it for Training can also remove their Search access. Agent has no Disallow option because Cloudflare says the web does not yet have a well-established directive for expressing that preference to agents. ⁹

How much of web traffic is now automated?

Defaults only matter at scale, and this one has it. Cloudflare fronts about a fifth of the web, automated traffic now generates roughly 57% of all web requests, and the mix inside crawler traffic has shifted hard toward training. ²⁵ The web is now something most of its visitors do not read.

Why does a network default matter more than robots.txt?

Because robots.txt asks and a network refuses. A disallow rule is a published request that well-behaved crawlers honour voluntarily; nothing in the protocol stops a crawler that ignores it. Independent log analysis by Oncrawl found AI crawlers (PerplexityBot, MistralAI-User and ClaudeBot among them) fetching URLs that were explicitly disallowed in robots.txt and receiving 200 and 304 responses, at volumes ranging from a few hundred to over a million requests a month on a single site. ⁸ Cloudflare’s categories are enforced at the edge instead, before the request reaches your origin. That is the difference between a sign on the door and a lock.

None of which makes the file pointless, and it is worth being precise about why rather than leaving a reader with "robots.txt is theatre". The specification says so itself: RFC 9309 describes its rules as not being a form of access authorization. The crawlers that decline are, on their operators’ own documentation, the user-initiated fetchers (a person asked for that page), while the scheduled crawlers that generate most of the volume do honour it. So the file still governs the traffic you most want governed, and it remains the only standard place to state a per-crawler policy at all. We work through what to write in it, and what has to sit behind it, in robots.txt in the AI era.

September 15 produced a hybrid rather than a simple network lock. For accountable mixed-use crawlers, Disallow AI Training publishes an operator-specific no-training preference and keeps Search available; Cloudflare blocks the other training crawlers. Choosing Block instead enforces denial at the edge and now includes mixed-use crawlers. The operational lesson survives: inspect the published preference and the responses bots actually receive, because a configured intention and an observed outcome are different evidence. ⁸⁹

Why did Cloudflare change the controls?

The old bargain was simple: crawl my site, send me clicks. It broke when answers started arriving without visits. Crawling stayed free; the traffic it used to pay with did not. Cloudflare’s answer is not only the block. Alongside it sits pay per crawl, which lets a publisher answer a crawler with HTTP 402 Payment Required and a price. The publisher sets Allow, Charge, or Block per crawler, and Cloudflare acts as merchant of record. It is a bilateral pricing mechanism rather than a marketplace, and as of 2026-09-21 Cloudflare’s current documentation says it remains in closed beta. ³¹⁰

What should a product brand do after September 15?

1. Record the actual migrated state. Existing legacy Block AI Bots settings were mapped into Search, Training, and Agent controls; “unchanged” is too loose a description even where Cloudflare preserved the practical effect. ⁹
2. For a new domain, check which onboarding preset was offered. Non-ad sites are offered Allow across all three controls; ad-monetized sites are offered Search: Allow, Training: Disallow AI Training, and Agent: Block on pages with ads. Every setting remains editable. ⁹
3. Make the category call deliberately. Keep Search available when discovery matters. Decide whether the ad-page Agent restriction fits the value of a shopper’s assistant reaching the page, and choose Disallow AI Training rather than Block when you want accountable mixed-use crawlers to retain Search access. ⁹
4. If Training bothers you, consider pricing it instead of blanket-blocking it. Pay per crawl exists for that trade, with the caveat that Cloudflare currently documents it as a closed beta. ³¹⁰
5. Verify from the outside, which is the step that survives any change of scope. The settings you believe you have and the responses bots actually receive routinely differ, so run a Hidden Layer audit to check live responses for 12 AI crawler user-agents, including the Search, Agent and Training split.
6. Re-check at the next domain launch, and again once Microsoft’s Bing no-training support (targeted for early 2027) ships. ⁹

Review, October 27, 2026: what happened after September 15?

This article promised a check after September 15. We made it on October 27, eleven days after the date we had set (October 16). The short result: the controls work as Cloudflare described them, and we cannot say whether they changed anyone’s AI visibility, because nobody, including us, has measured that. We had said we would define a visibility cohort by October 16. We did not, so that part of our promise was not kept.

What we could not use: a third-party crawler-blocking report updated October 1 shows GPTBot’s full-disallow share falling in late September, but it takes its numbers from Cloudflare Radar and attributes the dip to a change in which files were sampled, not to site owners changing their minds. It is not independent evidence and we do not rely on it. Cloudflare’s own claims that AI-referred visitors convert at three to five times the usual rate are unsourced and we make no claim from them. ¹²

Next review: March 31, 2027. By then Microsoft’s Bing support and Cloudflare’s AI-summary control, both targeted for early 2027, should have shipped or slipped, and we will check each against its primary source. A visibility cohort must be defined before that review, or the review will say again that the effect is untested. ⁹

What we know after launch is narrower and more useful than the prediction this article previously made: the controls shipped, but six weeks on there is still no post-deadline data that attributes a visibility change to them. Until a cohort is defined and measured, a claimed citation effect would be invented precision, not a finding. The next review is March 31, 2027.