Every AI crawler, explained.
The bots that decide what AI engines can read on your site — what each one does, its user-agent string, whether it honors robots.txt, and the exact rules to allow or block it.
The key distinction: training crawlers feed future models, search crawlers feed today's answers and citations. Blocking a training crawler is a principled opt-out; blocking a search crawler makes you invisible. Know which is which before you touch robots.txt.
Training crawlers
Collect web content to train AI models. Blocking these is a training opt-out — it doesn't change what AI engines say about you today.
GPTBot
OpenAI's training crawler. Blocking it opts you out of future model training — it doesn't affect ChatGPT citations.
ClaudeBot
Anthropic's training crawler. Replaced the retired anthropic-ai and Claude-Web tokens.
Bytespider
ByteDance's training crawler. Widely reported to ignore robots.txt — enforce at the firewall.
CCBot
Nonprofit Common Crawl bot. Blocking it is the broadest single training opt-out — it honors robots.txt.
Search & answer crawlers
Build the indexes behind AI answers and citations. Blocking these removes you from AI search results — the highest-cost block on this list.
OAI-SearchBot
Builds OpenAI's search index — the bot that gets you cited in ChatGPT Search.
Claude-SearchBot
Anthropic's search crawler — the bot that finds pages Claude can cite.
PerplexityBot
Crawls for Perplexity's answer engine, which cites sources inline aggressively.
User-triggered fetchers
Fetch a single page when a user explicitly asks about it. Some of these ignore robots.txt by design.
ChatGPT-User
Fetches pages users explicitly ask ChatGPT about. OpenAI says robots.txt rules may not apply to it.
Claude-User
Per-URL fetches for Claude users asking about a page. Honors robots.txt — the well-behaved user-fetcher.
Perplexity-User
User-triggered Perplexity fetches. Generally ignores robots.txt by design.
Control tokens (not crawlers)
Names you can reference in robots.txt to opt out of AI training use. They never fetch anything and never appear in your logs.
Google-Extended
Not a crawler — a robots.txt token opting your content out of Gemini training and grounding use.
Applebot-Extended
Not a crawler — a token opting your content out of Apple Intelligence training use.
Which bots actually obey robots.txt?
Honor it: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, CCBot. Ignore it (documented): Perplexity-User generally ignores it; ChatGPT-User's docs say rules "may not apply". Ignore it (reported): Bytespider is widely reported to disregard it. Google-Extended and Applebot-Extended aren't crawlers at all — they're opt-out tokens.
Blocking beyond robots.txt
robots.txt is a request, not a barrier. When a bot ignores it — or you want belt and suspenders — block by user agent at the server or CDN:
nginx
if ($http_user_agent ~* "Bytespider") { return 403; }Apache (.htaccess)
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule .* - [F,L]Cloudflare WAF
Create a custom rule: User Agent contains "Bytespider" → Block. Cloudflare also offers a one-click "Block AI bots" managed toggle under Security → Bots.
Check your own robots.txt
See which of these bots your site currently lets in — free, instant.
Published October 5, 2026. We re-verify these pages against operator documentation periodically; each bot page carries its own changelog.