CCBot: Common Crawl's Training-Data Crawler
Common Crawl · Training-data crawler (nonprofit) · Last verified October 2026
CCBot is the nonprofit Common Crawl's bot, building an open archive of the public web. Its corpus has trained a generation of language models, including early GPT versions — so blocking CCBot is arguably the broadest single training opt-out available. It honors robots.txt. Note: blocking stops future crawls, not use of snapshots already taken.
What is CCBot?
Nonprofit crawl of the public web. Its corpus has trained a generation of language models, including early GPT versions.
The user-agent string
CCBot/2.0 (+https://commoncrawl.org/faq/)
Does CCBot respect robots.txt?
Yes. CCBot is among the most polite, well-behaved crawlers — which is why it's the one most often blocked by accident in aggressive "block all AI" templates.
How to block CCBot in robots.txt
User-agent: CCBot Disallow: /
How to explicitly allow CCBot
Useful when your robots.txt has broad Disallow rules and you want to carve out an exception:
User-agent: CCBot Allow: /
One bot per User-agent block; specific tokens beat User-agent: *.
What blocking CCBot does (and doesn't do)
Blocking CCBot opts you out of future Common Crawl snapshots — and downstream, out of training sets built on them. It does not remove your content from existing archives or from models already trained on them. If your goal is only to opt out of corporate training while supporting open data, leave CCBot allowed and block the corporate tokens instead.
How to check if CCBot is crawling your site
Search your access logs for the token:
grep -i "CCBot" /var/log/nginx/access.log | wc -l
On Cloudflare, filter Firewall Events or Analytics by User Agent. If the count is zero over a month, the bot isn't visiting you — either your robots.txt blocks it, or it simply hasn't found you worth crawling.
Beyond robots.txt
robots.txt is a request, not a barrier. For server- and CDN-level blocking (nginx, Apache, Cloudflare WAF), see blocking beyond robots.txt on the overview page.
CCBot vs. related bots
Changelog
CCBot/2.0 is the current documented version. Last verified October 2026.
What does CCBot see on your site?
Operator docs: Common Crawl · All AI crawlers