Is Cloudflare blocking AI crawlers on my site?

4 min read. Published November 2026.

The short answer

It might be, and your robots.txt won't tell you. Cloudflare sits in front of your server. If it blocks a crawler, the request never reaches your site, so a robots.txt that says Allow: / changes nothing.

What changed

In September 2024 Cloudflare added a one-click setting to block AI crawlers. On July 1, 2025 it announced that blocking AI crawlers would become the default unless the site owner allows them, and that new domains signing up are asked whether to allow or block them. If you set up Cloudflare recently, or clicked through a setup screen without reading it, AI crawlers may be blocked on your site today.

Training bots and search bots are different

Blocking training crawlers such as GPTBot, ClaudeBot and CCBot keeps your content out of future model training. That's a fair choice, and it doesn't stop today's AI answers from citing you.

Blocking search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot is different. Those build the indexes AI answers cite, so blocking them takes you out of those answers. Make sure your setting matches what you actually want.

How to check

• In Cloudflare: look through your bot and AI crawler settings, and any custom firewall rules that match on user agent. Cloudflare has renamed this area more than once, so search the dashboard for "AI" or "bots".
• From the outside: our free grader sends a live request as each AI crawler and reports whether it got the page, was blocked or was challenged.
• By hand: send a request as the crawler yourself with one command.

In our test of 161 Fairfield County business sites, 18% (29 sites) refused or challenged at least one AI crawler at the server while their robots.txt allowed it. Cloudflare is one of the common causes.

Honest limits

Our live test sends each crawler's published user agent from our own server, not from OpenAI's or Perplexity's addresses. Cloudflare can treat a verified real bot differently from a request that only claims to be one. So a "challenged" result from us means "worth checking in your dashboard", not proof that the real bot is blocked.