AI Crawler Directory · Common Crawl
CCBot:
what it is and how to verify it.
CCBot is the crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data available to anyone. It is not owned by an AI company — but its dataset is the single most important training feedstock in the industry: most large language models trained before per-vendor crawlers existed learned the web substantially through Common Crawl snapshots.
That gives CCBot outsized leverage. Blocking it removes your content from a public dataset used by researchers, startups, and every AI lab that licenses or downloads Common Crawl — including many that run no crawler of their own. Allowing it is the widest possible distribution of your content into future models, one crawl for the whole ecosystem.
CCBot respects robots.txt and crawls from dedicated IP ranges with reverse-DNS verification, published as JSON. It fetches periodically to build monthly-cadence archive snapshots rather than continuously.
CCBot User-Agent string
CCBot/2.0 (https://commoncrawl.org/faq/)How to verify CCBot hits are real
The User-Agent header is plain text — any client can claim to be CCBot, and scrapers routinely do. Common Crawl crawls from dedicated ranges published at index.commoncrawl.org/ccbot.json, with reverse-DNS verification.
https://index.commoncrawl.org/ccbot.jsonFetch the list and check the hit’s source IP against it — for a quick manual look:
curl -s https://index.commoncrawl.org/ccbot.jsonAllow or block CCBot in robots.txt
This is a training-data control: allowing it lets your content shape what future models know; blocking it keeps your material out of training corpora without affecting live citations. A policy call, not a traffic one.
Allow
User-agent: CCBot
Allow: /Block
User-agent: CCBot
Disallow: /CCBot — frequently asked questions
What is CCBot?
CCBot is Common Crawl’s web crawler. Common Crawl is a non-profit that publishes an open repository of web crawl data used by researchers and AI developers worldwide. The bot identifies itself as CCBot/2.0.
Which AI models train on Common Crawl data?
Common Crawl does not publish a client list, but its archives are a documented major component of training corpora across the industry — historically including the datasets behind GPT-class and many open-source models. Blocking CCBot affects all downstream users at once.
Does CCBot respect robots.txt?
Yes. Adding "User-agent: CCBot / Disallow: /" to robots.txt keeps your site out of future Common Crawl snapshots. Existing archived snapshots are not retroactively deleted by a robots change.
How do I verify CCBot hits?
Common Crawl publishes its crawl IP ranges (IPv4 and IPv6) at index.commoncrawl.org/ccbot.json and supports reverse-DNS verification. A "CCBot" UA from outside those ranges is a spoof.
Should a small business block CCBot?
Usually not. For most companies, presence in the open corpus means AI assistants of every brand can learn your product facts. The main reasons to block are licensing-sensitive content or strict data-control policies.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $15/mo flat.
Start 7-day free trial — $0 due today7-day free trial · $15/mo · cancel anytime