AI crawler reference

GPTBot: OpenAI's Training Crawler

OpenAI · Training crawler · Last verified October 2026

GPTBot is OpenAI's training crawler. It crawls the public web to gather training data for future models — it does not power ChatGPT's answers or search citations. Its user-agent string is shown below. Block it with a User-agent: GPTBot + Disallow: / rule if you don't want your content in OpenAI's training data.

What is GPTBot?

Crawls the public web to gather training data for OpenAI's models. It does not answer user questions directly and does not build the search index ChatGPT cites.

The user-agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Does GPTBot respect robots.txt?

Yes. GPTBot honors robots.txt. OpenAI notes it can take about 24 hours for its systems to pick up a robots.txt change, so don't expect instant results after editing.

How to block GPTBot in robots.txt

User-agent: GPTBot
Disallow: /

How to explicitly allow GPTBot

Useful when your robots.txt has broad Disallow rules and you want to carve out an exception:

User-agent: GPTBot
Allow: /

One bot per User-agent block; specific tokens beat User-agent: *.

What blocking GPTBot does (and doesn't do)

Blocking GPTBot opts your content out of future OpenAI model training. It does not stop ChatGPT from citing your site today — that's OAI-SearchBot (search index) and ChatGPT-User (per-URL fetches), which are separate bots with separate rules. Blocking GPTBot alone keeps you fully visible in ChatGPT Search. This is the most misunderstood toggle in AI SEO.

How to check if GPTBot is crawling your site

Search your access logs for the token:

grep -i "GPTBot" /var/log/nginx/access.log | wc -l

On Cloudflare, filter Firewall Events or Analytics by User Agent. If the count is zero over a month, the bot isn't visiting you — either your robots.txt blocks it, or it simply hasn't found you worth crawling.

Beyond robots.txt

robots.txt is a request, not a barrier. For server- and CDN-level blocking (nginx, Apache, Cloudflare WAF), see blocking beyond robots.txt on the overview page.

GPTBot vs. related bots

OAI-SearchBot — the bot that actually gets you cited in ChatGPT Search — don't confuse the two.

ChatGPT-User — fetches single pages when users ask ChatGPT about a URL.

ClaudeBot — Anthropic's equivalent training crawler.

Changelog

GPTBot/1.4 is the current documented version — OpenAI warns the version number may change, so match on the GPTBot token, not the number. OpenAI split search crawling into OAI-SearchBot, so GPTBot no longer governs search visibility. Last verified October 2026.

What does GPTBot see on your site?

Check your robots.txt — free Grade my site — free

Operator docs: OpenAI · All AI crawlers