What we found
1. One in five sites blocks AI crawlers without saying so.
18.0% of graded sites (29/161) refused or challenged at least one AI crawler at the server level — while robots.txt said it was allowed in. A robots.txt-only checker calls every one of these sites “open.”
2. The real block rate is 6× what robots.txt shows.
Just 0.6% (1/161) of sites block an AI search crawler in robots.txt. Live testing puts the true number at 3.7% (6/161) — the rest are server-level blocks no robots.txt check can see. (A study of 3,014 Houston business sites published this month found the same pattern: 1.6% in robots.txt vs ~5% with server blocks counted.)
3. 12% of sites won’t serve a page to any bot at all.
31 of 251 sites refused our declared, benign fetcher outright — HTTP 403s and Cloudflare challenge walls — before we ever spoofed a crawler UA. If a plainly-identified bot can’t get the page, AI crawlers can’t either.
4. The average local business site grades C+ (72.7/100).
Only 9 of 161 reached A-range; 26 graded D or F. SEO averaged 75.3, AI-readiness 70.1.
5. The most common failures are AI-answer basics.
| Failed check | Sites | Share |
|---|---|---|
| No question-style headings | 132 | 82.0% |
| No llms.txt | 100 | 62.1% |
| No FAQ content | 97 | 60.2% |
| No structured data | 63 | 39.1% |
| Missing image alt text | 60 | 37.3% |
| H1 problems | 48 | 29.8% |
| Thin semantic HTML | 41 | 25.5% |
| Missing/weak meta description | 39 | 24.2% |
82% of sites have no question-style headings for AI answers to quote; 62% have no llms.txt; 60% have no FAQ content. These are afternoon fixes, not rebuilds.
Per-bot results
Robots.txt verdict vs. what our live request with each bot’s documented user-agent actually got back. “Server refused” = the page came back 401/403/429 or a bot-wall challenge page.
| Crawler | Blocked in robots.txt | Refused at server |
|---|---|---|
| GPTBot | 1 | 15 |
| ChatGPT-User | 1 | 5 |
| OAI-SearchBot | 1 | 5 |
| ClaudeBot | 1 | 18 |
| Claude-SearchBot | 1 | 5 |
| Claude-User | 1 | 4 |
| PerplexityBot | 1 | 5 |
| Perplexity-User | 1 | 4 |
| Google-Extended | 1 | 0 |
| CCBot | 1 | 22 |
Google-Extended never fetches (token only) and was not live-tested. One site blocked every crawler in robots.txt yet served the page to all of them anyway — robots.txt is a request some servers ignore.
Grade distribution
| Combined grade | Sites | Share |
|---|---|---|
| A-range (90+) | 9 | 5.6% |
| B (80-89) | 54 | 33.5% |
| C (60-79) | 72 | 44.7% |
| D/F (below 60) | 26 | 16.1% |
Methodology
- Site list: 251 unique business websites in Fairfield County, CT, compiled Oct 5–6, 2026 from the Greenwich Red Book trades directory and trade×town web searches. Nationals, franchises, and social-only presences were removed. Caveat: town coverage is uneven — Bridgeport, Monroe, Bethel, Brookfield, Easton, Weston, Redding, and New Fairfield are thin or absent.
- Testing: one full 19-check grade per site, plus one HTTP request per crawler using its documented user-agent string (9 crawlers, parallel, 12s timeout). One site at a time, ~3s between sites.
- Exclusions: 90 sites couldn’t be graded — 59 dead or broken (DNS failures, expired SSL, timeouts) and 31 that block automated fetching outright (counted in finding 3).
- Honesty note: a server can treat our user-agent-spoofed request differently from the real crawler, so treat live-fetch results as a signal, not proof. We did not bypass any bot wall and did not execute JavaScript — same plain-HTML policy as the grader.
- Raw data: data.csv — every graded site, all three scores, and per-bot robots.txt + live-fetch verdicts. Use it, cite it, argue with it.
See what AI engines can actually read on your site.
Free. 19 checks. 60 seconds. Including the live per-bot fetch test above.
Grade my site — free