robots.txt for AI crawlers, explained.

6 min read. Everything below is verifiable against the operators' own docs, linked at the bottom.

What robots.txt actually does

It's a plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which parts they're allowed to visit. That's it. It's a request, not a lock: well-behaved bots obey it, bad actors ignore it. Every major AI crawler claims to obey it, which makes it the cheapest AI-visibility lever you own.

The basic shape:

User-agent: GPTBot
Allow: /

User-agent: CCBot
Disallow: /private/

One block per bot. A specific bot's block beats the generic User-agent: * block. When rules conflict, the longest matching path wins.

The AI tokens you should know

There are three kinds, and mixing them up is the most common mistake we see in grades:

• Training crawlers gather data for future models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance). Blocking one opts your content out of future training. It does not stop today's ChatGPT from citing you.
• Retrieval and search crawlers find pages AI engines can cite right now: OAI-SearchBot, Claude-SearchBot, PerplexityBot. These are the ones that affect citations.
• User-triggered fetchers only visit when a user asks about a specific URL: ChatGPT-User, Claude-User, Perplexity-User. Blocking them breaks "summarize this page" features. Usually best left allowed.

Full reference for each, with user-agent strings: every AI crawler, explained.

The Google-Extended trap

Google-Extended is not a crawler. It's a control token that opts your content out of Google's AI model training. Blocking it does not affect Google Search rankings or AI Overviews. Letting it in does not help you rank. We see sites block it accidentally via copied templates, then wonder why nothing changed. Nothing changed because it was never doing what they thought.

How to check yours

Two free ways, no account:

• robots.txt checker — paste your URL, see per-bot allow/block for all 12 AI crawlers instantly.
• The full grader — 19 checks including per-bot robots.txt evaluation against your actual page path, not just the homepage.

Honest limits

robots.txt is voluntary. It won't stop scrapers that ignore it, and it can't fix content AI engines can't parse. It's one layer: necessary, not sufficient. Our grader checks the other layers too.

Operator docs: OpenAI bots · Anthropic · Perplexity · Google crawlers · Apple