AI crawler reference

ClaudeBot: Anthropic's Training Crawler

Anthropic · Training crawler · Last verified October 2026

ClaudeBot is Anthropic's training crawler, gathering web data for Claude models. Anthropic says it respects robots.txt and doesn't bypass paywalls or logins. Note: the older tokens anthropic-ai and Claude-Web are retired — a robots.txt that only names those no longer restricts anything.

What is ClaudeBot?

Crawls the web for training data used in Claude models. Anthropic says it respects robots.txt and doesn't bypass paywalls or logins.

The user-agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0

Does ClaudeBot respect robots.txt?

Yes, per Anthropic's published documentation.

How to block ClaudeBot in robots.txt

User-agent: ClaudeBot
Disallow: /

How to explicitly allow ClaudeBot

Useful when your robots.txt has broad Disallow rules and you want to carve out an exception:

User-agent: ClaudeBot
Allow: /

One bot per User-agent block; specific tokens beat User-agent: *.

What blocking ClaudeBot does (and doesn't do)

Blocking ClaudeBot opts your content out of Claude training data. It doesn't change what Claude says about you this week — current answers come from search crawlers and user fetchers, not training crawls. Same trade as GPTBot: a training opt-out, not a visibility change.

How to check if ClaudeBot is crawling your site

Search your access logs for the token:

grep -i "ClaudeBot" /var/log/nginx/access.log | wc -l

On Cloudflare, filter Firewall Events or Analytics by User Agent. If the count is zero over a month, the bot isn't visiting you — either your robots.txt blocks it, or it simply hasn't found you worth crawling.

Beyond robots.txt

robots.txt is a request, not a barrier. For server- and CDN-level blocking (nginx, Apache, Cloudflare WAF), see blocking beyond robots.txt on the overview page.

ClaudeBot vs. related bots

Claude-SearchBot — Anthropic's search crawler — the one behind Claude's citations.

Claude-User — per-URL fetches when Claude users ask about a page.

GPTBot — OpenAI's equivalent training crawler.

Changelog

ClaudeBot replaced the retired anthropic-ai and Claude-Web tokens; old robots.txt files naming only those are now ineffective. Last verified October 2026.

What does ClaudeBot see on your site?

Check your robots.txt — free Grade my site — free

Operator docs: Anthropic · All AI crawlers