AI crawler reference

OAI-SearchBot: OpenAI's Search Crawler

OpenAI · Search crawler · Last verified October 2026

OAI-SearchBot builds OpenAI's search index — the pages ChatGPT Search can cite. If ChatGPT Search never references you, this is the first bot to check. It honors robots.txt. Allow it explicitly with User-agent: OAI-SearchBot + Allow: /, especially if your robots.txt has broad Disallow rules that might catch it.

What is OAI-SearchBot?

Builds OpenAI's search index — the pages ChatGPT Search can cite. This is the crawler that most directly affects whether ChatGPT references you.

The user-agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

Does OAI-SearchBot respect robots.txt?

Yes, OAI-SearchBot honors robots.txt.

How to block OAI-SearchBot in robots.txt

User-agent: OAI-SearchBot
Disallow: /

How to explicitly allow OAI-SearchBot

Useful when your robots.txt has broad Disallow rules and you want to carve out an exception:

User-agent: OAI-SearchBot
Allow: /

One bot per User-agent block; specific tokens beat User-agent: *.

What blocking OAI-SearchBot does (and doesn't do)

Blocking OAI-SearchBot removes you from OpenAI's search index: ChatGPT Search can't cite pages it can't index. This is the highest-cost block on this list for AI visibility. Don't confuse it with GPTBot (training only) — blocking GPTBot is a training opt-out, blocking OAI-SearchBot is a visibility opt-out.

How to check if OAI-SearchBot is crawling your site

Search your access logs for the token:

grep -i "OAI-SearchBot" /var/log/nginx/access.log | wc -l

On Cloudflare, filter Firewall Events or Analytics by User Agent. If the count is zero over a month, the bot isn't visiting you — either your robots.txt blocks it, or it simply hasn't found you worth crawling.

Beyond robots.txt

robots.txt is a request, not a barrier. For server- and CDN-level blocking (nginx, Apache, Cloudflare WAF), see blocking beyond robots.txt on the overview page.

OAI-SearchBot vs. related bots

GPTBot — OpenAI's training crawler — often confused with this bot.

ChatGPT-User — fetches single pages on user request.

PerplexityBot — Perplexity's equivalent search crawler.

Changelog

OpenAI introduced OAI-SearchBot as its dedicated search index crawler; GPTBot no longer governs search. Last verified October 2026.

What does OAI-SearchBot see on your site?

Check your robots.txt — free Grade my site — free

Operator docs: OpenAI · All AI crawlers