AI crawler reference

Claude-SearchBot: Anthropic's Search Crawler

Anthropic · Search crawler · Last verified October 2026

Claude-SearchBot is Anthropic's crawler for search and retrieval features — the one that finds pages Claude can cite. If you want Claude's search features to see your content, this is the bot to allow. It honors robots.txt.

What is Claude-SearchBot?

Anthropic's crawler for search and retrieval features — the one that finds pages Claude can cite.

The user-agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-SearchBot/1.0

Does Claude-SearchBot respect robots.txt?

Yes, per Anthropic's documentation.

How to block Claude-SearchBot in robots.txt

User-agent: Claude-SearchBot
Disallow: /

How to explicitly allow Claude-SearchBot

Useful when your robots.txt has broad Disallow rules and you want to carve out an exception:

User-agent: Claude-SearchBot
Allow: /

One bot per User-agent block; specific tokens beat User-agent: *.

What blocking Claude-SearchBot does (and doesn't do)

Blocking Claude-SearchBot hides your pages from Claude's search-driven citations. Allow it unless you want to be invisible in Claude's search features. Don't confuse it with ClaudeBot (training data) — they serve completely different purposes.

How to check if Claude-SearchBot is crawling your site

Search your access logs for the token:

grep -i "Claude-SearchBot" /var/log/nginx/access.log | wc -l

On Cloudflare, filter Firewall Events or Analytics by User Agent. If the count is zero over a month, the bot isn't visiting you — either your robots.txt blocks it, or it simply hasn't found you worth crawling.

Beyond robots.txt

robots.txt is a request, not a barrier. For server- and CDN-level blocking (nginx, Apache, Cloudflare WAF), see blocking beyond robots.txt on the overview page.

Claude-SearchBot vs. related bots

ClaudeBot — Anthropic's training crawler.

Claude-User — fetches single pages on user request.

OAI-SearchBot — OpenAI's equivalent search crawler.

Changelog

Anthropic's newer search crawler; if it stops appearing in your logs, check their docs for the current token name. Last verified October 2026.

What does Claude-SearchBot see on your site?

Check your robots.txt — free Grade my site — free

Operator docs: Anthropic · All AI crawlers