AI crawler reference

ChatGPT-User: OpenAI's On-Demand Page Fetcher

OpenAI · User-triggered fetcher · Last verified October 2026

ChatGPT-User fetches a single page only when a ChatGPT user explicitly asks about it ("summarize this URL"). It doesn't crawl on its own. OpenAI documents that robots.txt rules may not apply to it, since a person — not a crawler — triggered the request. A robots.txt block gives a false sense of control here.

What is ChatGPT-User?

Fetches a specific page only when a ChatGPT user explicitly asks about it. It doesn't crawl the web independently.

The user-agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot.json

Does ChatGPT-User respect robots.txt?

Unreliable. OpenAI's documentation says robots.txt rules "may not apply" to ChatGPT-User because the fetch is user-initiated rather than automatic crawling. Treat a robots.txt block as a request it may ignore.

How to block ChatGPT-User in robots.txt

User-agent: ChatGPT-User
Disallow: /

Important: ChatGPT-User is documented as potentially ignoring robots.txt (see above). This rule is still worth writing — it signals intent and stops compliant variants — but don't rely on it alone.

How to explicitly allow ChatGPT-User

Useful when your robots.txt has broad Disallow rules and you want to carve out an exception:

User-agent: ChatGPT-User
Allow: /

One bot per User-agent block; specific tokens beat User-agent: *.

What blocking ChatGPT-User does (and doesn't do)

If the block is honored, users can't ask ChatGPT to read your pages. If it's ignored (documented behavior), nothing changes. Either way, blocking ChatGPT-User doesn't affect whether ChatGPT Search cites you — that's OAI-SearchBot's job. Most sites should allow it; blocking mainly breaks the "read this page for me" feature without reliably stopping fetches.

How to check if ChatGPT-User is crawling your site

Search your access logs for the token:

grep -i "ChatGPT-User" /var/log/nginx/access.log | wc -l

On Cloudflare, filter Firewall Events or Analytics by User Agent. If the count is zero over a month, the bot isn't visiting you — either your robots.txt blocks it, or it simply hasn't found you worth crawling.

Beyond robots.txt

robots.txt is a request, not a barrier. For server- and CDN-level blocking (nginx, Apache, Cloudflare WAF), see blocking beyond robots.txt on the overview page.

ChatGPT-User vs. related bots

GPTBot — OpenAI's training crawler — a different bot with a different purpose.

OAI-SearchBot — builds the search index behind ChatGPT citations.

Perplexity-User — the same user-fetcher pattern at Perplexity — also ignores robots.txt.

Changelog

Last verified October 2026.

What does ChatGPT-User see on your site?

Check your robots.txt — free Grade my site — free

Operator docs: OpenAI · All AI crawlers