How to check if GPTBot and other AI crawlers can reach your site

5 min read. Published November 2026.

Two layers, two checks

A crawler needs permission and a page. robots.txt gives permission. Your server, CDN or firewall decides whether the crawler actually gets the page. Most guides only check the first layer.

1. Check robots.txt

Open yoursite.com/robots.txt. Look for a block that names the crawler, such as User-agent: GPTBot. If there isn't one, the User-agent: * block applies. Or paste your URL into the robots.txt checker to see all 12 AI crawlers at once. How robots.txt works for AI crawlers.

2. Ask your server as the crawler

Run this in a terminal (Mac and Linux have curl built in). It prints the HTTP status your server returns to GPTBot's user agent:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://yoursite.com/

Then run it again without the -A line. If the normal request gets 200 and the crawler gets 403, 429 or 503, something in front of your site is turning the crawler away.

Swap in these user agents for the crawlers that matter most:

3. Look in your server logs

If you can see your access logs, search them for these crawler names. Real visits that end in 403 or 429 are the clearest sign of a block.

Honest limits

A request from your laptop only claims to be GPTBot. Some firewalls verify the real bot by its network address and let it in while blocking imitations, and some do the reverse. Treat a mismatch as a reason to check your firewall or CDN settings, not as a final answer. In our test of 161 Fairfield County business sites, 18% (29 sites) refused or challenged at least one AI crawler at the server while their robots.txt allowed it.