Making your site readable by AI engines.
Plain-language guides. No jargon, no fake metrics, everything verifiable. New guides as the crawlers change.
robots.txt for AI crawlers, explained
What robots.txt actually does, the AI bot tokens you should know, the Google-Extended trap, and how to check yours. 6 min read.
Every AI crawler, explained
Reference pages for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more. User-agent strings and allow/block snippets.
Is Cloudflare blocking AI crawlers on my site?
Cloudflare can turn AI crawlers away before robots.txt is ever read. How to tell, and which bots to let through. 4 min read.
How to check if GPTBot and other AI crawlers can reach your site
Two checks, one command: what robots.txt allows, and what your server actually sends each AI crawler. 5 min read.
What is llms.txt, and do you need one?
A proposed map of your site for AI models. Who says they read it (almost nobody yet), and how to write one in ten minutes. 4 min read.
robots.txt says allowed. Why can't AI crawlers read my site?
The usual reasons a crawler is allowed on paper but turned away in practice, and how to find which one you have. 5 min read.
Research: 251 Fairfield County business sites, tested for AI readability
Almost 1 in 5 block AI crawlers at the server while robots.txt says they're welcome. Method and raw data included.
Reading is good. Measuring is better.
Guides tell you what to do. The grader tells you what you actually did. Free, 60 seconds, no account.