CCBot:
what it is and how to verify it.

CCBot is the crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data available to anyone. It is not owned by an AI company — but its dataset is the single most important training feedstock in the industry: most large language models trained before per-vendor crawlers existed learned the web substantially through Common Crawl snapshots.

That gives CCBot outsized leverage. Blocking it removes your content from a public dataset used by researchers, startups, and every AI lab that licenses or downloads Common Crawl — including many that run no crawler of their own. Allowing it is the widest possible distribution of your content into future models, one crawl for the whole ecosystem.

CCBot respects robots.txt and crawls from dedicated IP ranges with reverse-DNS verification, published as JSON. It fetches periodically to build monthly-cadence archive snapshots rather than continuously.