Cloudflare Flips the Default: AI Crawlers Get Blocked on September 15 Unless You Decide Otherwise

On September 15, 2026, Cloudflare changes what happens when an AI system visits a website that never expressed an opinion about AI. For all new domains onboarding after that date, bots classified as Training or Agent are blocked by default on pages that display ads. Search crawlers stay allowed, and existing customers keep whatever settings they already have. ¹

That scope is narrower than the headlines suggest, and it is the reason this matters rather than the reason it does not. A platform default is a statement about which way the ratchet turns. Cloudflare has now shipped the three-category control to every customer and set the new-domain default to deny two of them. Nobody has to touch your existing configuration for that to change what launching a site looks like.

What are the three bot categories, and what does blocking each one cost?

The September change rides on a taxonomy Cloudflare introduced on July 1: every AI crawler request is classified as Search, Agent, or Training, and each category can be allowed or blocked independently. ¹ The names matter less than the trade each one represents. Three categories, three separate bets, one checkbox most people have never opened.

The three categories fail differently. Block Search and you disappear from AI search products. Block Agent and the assistant a shopper is using cannot read your product page at the exact moment it is deciding what to recommend. Block Training and future models simply know less about you. That is the slowest and least visible consequence of the three.

How much of web traffic is now automated?

Defaults only matter at scale, and this one has it. Cloudflare fronts about a fifth of the web, automated traffic now generates roughly 57% of all web requests, and the mix inside crawler traffic has shifted hard toward training. ²⁵ The web is now something most of its visitors do not read.

Why does a network default matter more than robots.txt?

Because robots.txt asks and a network refuses. A disallow rule is a published request that well-behaved crawlers honour voluntarily; nothing in the protocol stops a crawler that ignores it. Independent log analysis by Oncrawl found AI crawlers (PerplexityBot, MistralAI-User and ClaudeBot among them) fetching URLs that were explicitly disallowed in robots.txt and receiving 200 and 304 responses, at volumes ranging from a few hundred to over a million requests a month on a single site. ⁸ Cloudflare’s categories are enforced at the edge instead, before the request reaches your origin. That is the difference between a sign on the door and a lock.

None of which makes the file pointless, and it is worth being precise about why rather than leaving a reader with "robots.txt is theatre". The specification says so itself: RFC 9309 describes its rules as not being a form of access authorization. The crawlers that decline are, on their operators’ own documentation, the user-initiated fetchers — a person asked for that page — while the scheduled crawlers that generate most of the volume do honour it. So the file still governs the traffic you most want governed, and it remains the only standard place to state a per-crawler policy at all. We work through what to write in it, and what has to sit behind it, in robots.txt in the AI era.

This is what makes September 15 worth more than a diary entry. A default written into a file that crawlers may disregard is advisory. A default written into the network that fronts roughly a fifth of the web is operative. Two consequences follow for anyone reading this. First, the settings you believe you have published are not necessarily the settings being obeyed, and the only way to establish which is to look at what bots actually receive. Second, our own crawler policy, published per-category and deliberately permissive, depends on exactly the voluntary compliance this section says is unreliable. We publish it anyway, and we check the responses rather than trusting the file. ⁸

Why did Cloudflare flip the default?

The old bargain was simple: crawl my site, send me clicks. It broke when answers started arriving without visits. Crawling stayed free; the traffic it used to pay with did not. Cloudflare’s answer is not only the block. Alongside it sits pay per crawl, which lets a publisher answer a crawler with HTTP 402 Payment Required and a price. The publisher sets Allow, Charge, or Block per crawler, and Cloudflare acts as merchant of record. It is a bilateral pricing mechanism rather than a marketplace, and as of 2026-08-13 it remains in private beta, which is a long time to leave the only paid alternative to a block behind a waitlist. ³

What should a product brand do before September 15?

1. Establish exposure honestly. The new defaults apply to domains onboarding to Cloudflare after September 15, on pages that display ads. If every domain you own already exists on Cloudflare, the flip does not touch you and only step one stops applying. The rest is the same work it was last week. ¹
2. Make the category call deliberately, whether or not the deadline applies to you. Allow Search. Allow Agent, because assistants acting for shoppers fetch in real time and that is your buyer. Decide Training as a policy question rather than by inheriting whatever your account was created with.
3. If Training bothers you, price it instead of blanket-blocking it. Pay per crawl exists for that trade, with the caveat that it is still in private beta. ³
4. Verify from the outside, which is the step that survives any change of scope. The settings you believe you have and the responses bots actually receive routinely differ, so run a Hidden Layer audit to check live responses for 12 AI crawler user-agents, including the Search, Agent and Training split.
5. Re-check after September 15, and again at your next domain launch. Defaults applied by infrastructure are how sites go dark by accident: no error, no alert, just absence.

What we will watch next: whether Agent-category blocking measurably dents how often assistants cite affected domains after September 15. That is a testable prediction, and our visibility panel is built to test it. If the defaults bite, the sites that opted in deliberately will be the control group everyone else wishes they were in.