robots.txt in the AI era: what it controls, and what it only asks

Martijn Koster proposed the Robots Exclusion Protocol in 1994 as a way for a site owner to tell a search crawler which paths to skip. For most of the next thirty years it governed a handful of well-behaved crawlers, and almost nobody had to think about what would happen if one of them declined. In September 2022 the IETF published it as RFC 9309, a Standards Track document, which is the moment it stopped being a convention and became a specification. ¹

That specification is unusually candid about its own limits. It says in plain terms that the rules "are not a form of access authorization", and it describes crawlers as being requested to honour them. The protocol that the entire AI-crawler debate runs on documents, in its own text, that it cannot compel anyone. ¹

Which AI crawlers say they obey robots.txt, and which say they do not?

This is answerable without a single log file, because the operators publish it. The answer redraws the map. The usual split people reach for is training crawlers versus search crawlers, on the theory that training is the invasive one. That is not the line the operators themselves draw. The line is whether a request was generated by a schedule or by a person.

Read as of 18 August 2026, and read narrowly. What follows is three operators and eight crawlers, chosen because each publishes a per-crawler policy precise enough to quote. It is not a census: Google, Meta, Apple, Cohere, Mistral and ByteDance all run crawlers with their own tokens and their own terms, and a few of them say considerably less about compliance than the three below. Bot documentation also changes without announcement, which is why this page carries a review date rather than a claim to be current forever.

Two of the three operators here document a fetcher that steps outside the file, and both give the same reason: a person asked for that specific page, so the fetch is treated as browsing rather than crawling. Anthropic documents no such exception and states that all three of its bots honour the directives. That difference is a genuine policy divergence between operators, published by them, and it is worth more to a reader than any aggregate.

If it is only a request, why publish the file at all?

Because the crawlers that honour it are, for most brands, the ones that matter commercially. A training crawler you would rather not feed is the one you can actually turn away. A search crawler that sends an assistant’s user to your page is the one you want, and the file is the only standard place to say so per crawler. Silence is not neutrality: an absent rule falls through to the wildcard group, which on most sites is a blanket allow, and on Cloudflare-fronted domains onboarding after September 15, 2026 falls the other way. ⁹

It is also the file every audit, every vendor dashboard and every compliance reviewer reads first. Whatever it can and cannot enforce, it is the public record of what you asked for, and there is no second place to put that.

How should the file be written?

One group per crawler, named. Most AI crawlers look for their own user-agent token and fall back to the wildcard group only when they do not find one, so an unnamed crawler inherits whatever you wrote for everything else. Split by what the crawler is for rather than by which company sent it, because the trade is different in each direction: training is content absorbed permanently into weights, search and retrieval is distribution back to you.

# Training crawlers — a policy decision, not a default.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Search and retrieval — this is AI-era distribution. Keep it open.
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Note what this file does not do. It says nothing to ChatGPT-User or Perplexity-User, and adding groups for them would not change what those fetchers do, because both operators have published that a user-initiated fetch is treated differently. If you need those requests stopped, robots.txt is the wrong instrument and the next section is about the right one. ²³

What is Content-Signal, and does it change anything?

Cloudflare published the Content Signals Policy on 24 September 2025. It adds a directive inside a robots.txt group that separates what a crawler may DO with content from whether it may fetch it: search, ai-input and ai-train, each set to yes or no. Allow and Disallow govern access; Content-Signal governs use. ⁵

User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

It is a real directive line inside a group, not a comment, and omitting a signal grants and forbids nothing for that use. On enforcement Cloudflare is explicit and worth quoting: content signals "express preferences; they are not technical countermeasures", and its own guidance is to pair them with WAF rules and bot management if you want them to bind. A stated preference is still worth stating, because it is evidence of intent that a court or a counterparty can read. It is not a control. ⁵

So what actually enforces a crawler policy?

Something that can refuse the request. robots.txt is read by the crawler and obeyed at its discretion; an edge rule is applied before the request reaches your origin, and the crawler’s discretion never enters into it. That is the difference between a sign on the door and a lock, and it is why the September 15 Cloudflare default change matters more than any robots.txt edit you have ever made. We covered that change and the per-category decision behind it in Cloudflare Flips the Default. ⁹

Identity is the other half, and it is the part most teams skip. A user-agent string is a claim, not a credential: anything can send "GPTBot". Cloudflare’s verified-bot programme exists to close that gap, and its criteria are worth reading as a definition of good behaviour rather than as a vendor feature. A bot qualifies by identifying itself honestly, through cryptographic signatures, published IP ranges with stable user-agents, or reverse DNS, and by behaving non-abusively, which explicitly includes respecting robots.txt. Verification is how "this request says it is GPTBot" becomes "this request is GPTBot". ⁶

There is a failure mode in the other direction, and it is the one we see most often. Edge bot-blocking that is tuned to stop non-browser traffic will also stop the crawlers you wanted. When a crawler requests robots.txt and receives a 403, it cannot distinguish a missing file from a hostile CDN, and the practical result is a blanket disallow you never wrote. The fix is not to loosen the rules but to allow verified bots, then check the result from outside rather than in the dashboard. ⁶

# What a crawler actually receives. Run it from OUTSIDE your network.
for ua in GPTBot ClaudeBot PerplexityBot OAI-SearchBot Claude-SearchBot; do
  printf '%-20s %s\n' "$ua" \
    "$(curl -sS -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/robots.txt)"
done

# 200 = reachable. 403 = your edge is answering for you, and every rule
# in the file below is moot for that crawler.

A 403 there is the answer to a question most teams never ask, and it outranks everything else on this page: a crawler that cannot read the file cannot obey it. Check the same way after any WAF or bot-management change, because that is when this breaks and nothing alerts you.

What should you actually do?

1. Write the file per crawler, named, split by purpose rather than by vendor. Training is a policy decision; search and retrieval are distribution. An unnamed crawler inherits your wildcard group, so silence is a decision you did not know you made. ¹
2. Add Content-Signal to the wildcard group if you want your intent on the record, with the correct syntax and inside the group rather than as a trailing comment. It states a preference; it does not enforce one. ⁵
3. Accept that user-initiated fetchers are outside the file. Two of the three major operators document it. If those requests genuinely need stopping, that decision belongs at the edge, not in robots.txt. ²³
4. Verify from the outside, with the loop above, then check your server logs for disallowed paths returning 200. The settings you believe you published and the responses bots actually receive routinely differ — run a Hidden Layer audit to check live responses for 12 AI crawler user-agents in one pass. ⁷
5. Put enforcement where refusal is possible. Allow verified bots at the edge so honest crawlers are not caught by bot-fight rules, and treat the per-category controls as the real policy surface. ⁶⁹

The honest summary is that robots.txt is necessary, insufficient, and still worth doing well. It is the public record of your intent, it reliably governs the automated crawlers that make up most of the volume, and it is read by every auditor who looks at you. What it is not is a boundary. Treat it as the statement of policy and put the enforcement somewhere a request can be refused, and you have the arrangement the specification actually describes rather than the one most teams assume they have. ¹