Hidden Layer/Research/The 12 AI bots crawling your site right now
RENDERDOMSTYLESCHEMANETWORKBOT REGISTRY

Bot Registry

The 12 AI bots crawling your site right now

A field guide to every major AI crawler: what company runs it, what it does with your content, whether it respects robots.txt, and what UA string to put in your rules.

HP
Harshak Patel
Founder & Head of Research, Hidden Layer
3 min read
On this page

There are twelve AI crawlers that matter right now. Some train models. Some power live search. Some fetch content on-demand when a user asks an AI tool to read a URL. All of them use your robots.txt as policy — if you have one.

Training crawlers

Training crawlers download content at scale for use in model training datasets. They're not tied to live user queries — they run on schedules, build large corpora, and the content they collect shows up in model behaviour months later.

GPTBot — OpenAI

UA: Mozilla/5.0 AppleWebKit/537.36 (compatible; GPTBot/1.2; +https://openai.com/gptbot)

OpenAI's primary training crawler. Respects robots.txt. Blocking GPTBot means your content doesn't appear in future GPT training runs — it doesn't affect existing model knowledge or live ChatGPT browsing.

ClaudeBot — Anthropic

UA: Mozilla/5.0 AppleWebKit/537.36 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Anthropic's training crawler. Respects robots.txt. Like GPTBot, blocking it affects future training, not current model knowledge.

Google-Extended — Google

Separate from Googlebot (search indexing). Controls whether your content trains Bard/Gemini. Can be blocked independently without affecting Google Search.

Applebot-Extended — Apple

Apple's AI training crawler, used for training Apple Intelligence models. Distinct from Applebot (used for Spotlight and Siri search results).

CCBot — Common Crawl

Non-profit crawler that publishes a public web archive. Many open-source models train on Common Crawl data. Frequently blocked by brands concerned about AI training.

Meta-ExternalAgent — Meta

Used for training Meta's Llama and other AI models. Respects robots.txt.

Bytespider — ByteDance

ByteDance's crawler, used for TikTok, Doubao, and other AI products. Historically controversial for its robots.txt compliance. Hidden Layer recommends blocking by default unless you have specific reason to allow ByteDance AI training.

Retrieval/search crawlers

These bots fetch content in response to live user queries — when you ask Perplexity a question, it sends PerplexityBot to retrieve current information.

OAI-SearchBot — OpenAI

Powers ChatGPT's web browsing and SearchGPT. This is the bot you want to allow even if you block GPTBot — it sends users to your site and appears as a referral source in analytics.

Claude-SearchBot — Anthropic

Powers Claude's web search feature. Same commercial reasoning as OAI-SearchBot — this is traffic and discovery, not training.

PerplexityBot — Perplexity

Powers Perplexity's real-time answers. Blocking PerplexityBot means your content doesn't appear in Perplexity answers, which for some industries is now a meaningful traffic source.

On-demand fetch bots

ChatGPT-User — OpenAI

Used when a ChatGPT user shares a URL in conversation. Appears as a session visit, not a continuous crawl.

Claude-User — Anthropic

When a user pastes a URL and asks Claude to read it, this is the UA string used.

The robots.txt template

For most brands, the recommended starting point:

DIRECTIVE
# Training crawlers — allow by default
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

# Training crawlers — common to block
User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Retrieval bots — almost always allow
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# On-demand fetch — almost always allow
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

This makes your intent explicit rather than relying on the User-agent: * fallback. Update it when new AI crawlers launch — this list will grow.

AI BotsBot Registryrobots.txt
ShareLinkedInXEmail

Continue reading

News Analysis
Cloudflare Flips the Default: AI Crawlers Get Blocked on September 15 Unless You Decide Otherwise

From September 15 the network in front of about a fifth of the web blocks AI training and agent traffic by default on ad-supported pages, for every domain onboarding after that date. Existing settings are untouched, which is the part worth reading twice.

News Analysis
Google made GEO “official”: what the docs actually say

Google now tells you how to optimize for AI search. The honest reading: Google declared GEO to be SEO and told you to stop buying AI files and special schema. That advice is also self-serving, so we audit Google’s claims the same way we audit everyone’s.

Citation Intelligence
Ghost citations: why AI cited your content but forgot your brand

A source link in an AI answer is not the same as a brand recommendation. As of 2026-06-18, the cleaner metric is ghost citation rate: cited without being named.

Cite this article

Full
Harshak Patel. “The 12 AI bots crawling your site right now.” Hidden Layer, 8 April 2026. https://hidden-layer-blogs.pages.dev/post/ai-bot-registry
In line
Hidden Layer (2026)

Reuse

Republish, translate, excerpt or adapt the article text and the figures Hidden Layer drew under Creative Commons Attribution 4.0 International (CC BY 4.0), provided you credit Hidden Layer and link to the original. Read the CC BY 4.0 terms.

The grant covers our own words and charts only. It does not extend to data and figures quoted from other organisations, which stay with their owners; to trademarks and logos, ours and everyone else’s; or to audit reports and customer data produced by the product.

Author

HP
Harshak PatelFounder & Head of Research, Hidden Layer
Harshak Patel runs Hidden Layer, where the work is auditing how AI systems surface — or refuse to surface — brands and products. Background in enterprise product data and catalogue intelligence. The publishing rule here is simple: every article ships with its sources, its per-fact confidence, and the claims that were cut. The methodology is public and reproducible, and that, not the byline, is the credential.

Next

See how your domain scores against these checks.

Run a free audit

GEO Week — every Friday

Weekly brief on AI discoverability, agent readiness, and what shipped in the GEO space. No fluff.

We'll never spam you. Unsubscribe anytime. GDPR-compliant double opt-in.

← All research