robots.txt says allowed. Why can't AI crawlers read my site?

5 min read. Published November 2026.

Allowed is not the same as reachable

robots.txt is a sign on the door. It doesn't open the door. In our test of 161 Fairfield County business sites, 18% (29 sites) refused or challenged at least one AI crawler at the server while their robots.txt allowed it. Only 1 of those sites blocked an AI search crawler in robots.txt, but live testing found 6 that actually did.

The usual causes

• CDN bot protection. Services like Cloudflare can block or challenge AI crawlers before your server sees them. How to check Cloudflare.
• Firewall rules. A rule that blocks unfamiliar user agents, or anything with "bot" in its name, catches AI crawlers too.
• Security plugins and hosting firewalls. Some block automated visitors by default. Look for bot or crawler settings.
• Rate limits. A crawler that requests several pages quickly can get 429 errors and give up.
• Pages built only in JavaScript. Several AI crawlers have been reported not to run JavaScript. If your text only appears after scripts run, they may get an empty page even with a 200 status.

How to find yours

Send a request as the crawler and compare the status code with a normal request. A 403 or challenge points to the CDN or firewall. A 429 points to rate limits. A 200 with little text points to JavaScript. If you're not sure, ask your host which bot protection is on.

What to change

Allow the specific crawlers you want, not every bot. Most sites want the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the user-triggered fetchers, and can decide either way on training crawlers. Every AI crawler, explained.