AI crawler reference

Claude-User: Anthropic's On-Demand Page Fetcher

Anthropic · User-triggered fetcher · Last verified October 2026

Claude-User is the token for fetches Claude makes on behalf of a user asking about a specific page. Unlike OpenAI's and Perplexity's user-fetchers, Anthropic states that Claude-User honors robots.txt. Blocking it can break "read this URL" style features in Claude, so allow it unless you have a specific reason not to.

What is Claude-User?

Reported token for fetches Claude makes on behalf of a user asking about a specific page.

The user-agent string

Claude-User/1.0

Does Claude-User respect robots.txt?

Yes — Anthropic documents all three of its bots, including Claude-User, as honoring robots.txt. That makes it the exception among user-triggered fetchers.

How to block Claude-User in robots.txt

User-agent: Claude-User
Disallow: /

How to explicitly allow Claude-User

Useful when your robots.txt has broad Disallow rules and you want to carve out an exception:

User-agent: Claude-User
Allow: /

One bot per User-agent block; specific tokens beat User-agent: *.

What blocking Claude-User does (and doesn't do)

Blocking Claude-User stops Claude from reading specific pages users ask about. It doesn't affect Claude's general knowledge or its search citations. Since the block is actually honored here (unlike ChatGPT-User), the trade-off is real: privacy/control vs. breaking a user-facing feature.

How to check if Claude-User is crawling your site

Search your access logs for the token:

grep -i "Claude-User" /var/log/nginx/access.log | wc -l

On Cloudflare, filter Firewall Events or Analytics by User Agent. If the count is zero over a month, the bot isn't visiting you — either your robots.txt blocks it, or it simply hasn't found you worth crawling.

Beyond robots.txt

robots.txt is a request, not a barrier. For server- and CDN-level blocking (nginx, Apache, Cloudflare WAF), see blocking beyond robots.txt on the overview page.

Claude-User vs. related bots

ClaudeBot — Anthropic's training crawler.

Claude-SearchBot — Anthropic's search crawler.

ChatGPT-User — OpenAI's equivalent — but that one may ignore robots.txt.

Changelog

Less widely documented than ClaudeBot; included because it appears in real-world robots.txt files and server logs. Last verified October 2026.

What does Claude-User see on your site?

Check your robots.txt — free Grade my site — free

Operator docs: Anthropic · All AI crawlers