AI Crawler Directory · Anthropic
ClaudeBot:
what it is and how to verify it.
OperatorAnthropic
PurposeAI model training
Respects robots.txtYes
Executes JavaScriptNo
Hits verifiableYes — published IPs
TypeHTTP crawler / fetcher
ClaudeBot is Anthropic’s training crawler. It collects publicly available web content used to train and improve Claude foundation models. Like GPTBot for OpenAI, it works on its own crawl schedule and has no connection to any live user session — a ClaudeBot hit is corpus collection, not a reader.
Anthropic commits to transparent, non-disruptive crawling: ClaudeBot respects robots.txt directives (including Crawl-delay) and anti-circumvention measures like CAPTCHAs. Disallowing it signals that your future content should be excluded from training datasets.
The trade-off mirrors GPTBot: allowing ClaudeBot lets your product facts, docs, and expertise shape what Claude models know natively; blocking it keeps content out of training but does not affect whether Claude’s live search cites you — that is Claude-SearchBot’s job.
ClaudeBot User-Agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)Representative format — Anthropic does not publish an exact canonical UA string; the bot token is the stable identifier.
How to verify ClaudeBot hits are real
The User-Agent header is plain text — any client can claim to be ClaudeBot, and scrapers routinely do. Anthropic publishes verified crawler IPs for all its bots at claude.com/crawling/bots.json.
https://claude.com/crawling/bots.jsonFetch the list and check the hit’s source IP against it — for a quick manual look:
curl -s https://claude.com/crawling/bots.jsonAllow or block ClaudeBot in robots.txt
This is a training-data control: allowing it lets your content shape what future models know; blocking it keeps your material out of training corpora without affecting live citations. A policy call, not a traffic one.
Allow
User-agent: ClaudeBot
Allow: /Block
User-agent: ClaudeBot
Disallow: /ClaudeBot — frequently asked questions
What is ClaudeBot?
ClaudeBot is Anthropic’s web crawler for collecting publicly available content that may be used to train Claude AI models. It identifies itself with the ClaudeBot token in the User-Agent header and respects robots.txt.
Does ClaudeBot respect robots.txt?
Yes. Anthropic documents that ClaudeBot honors robots.txt disallow rules and the Crawl-delay extension, and states its crawlers do not bypass CAPTCHAs or similar anti-crawling measures.
How do I verify a ClaudeBot hit is really Anthropic?
Check the source IP against Anthropic’s published list at claude.com/crawling/bots.json. UA spoofing is common — several scraping tools ship "ClaudeBot" as a preset — so only an IP match confirms authenticity.
Will blocking ClaudeBot remove me from Claude’s answers?
Not from live, search-backed answers — those come via Claude-SearchBot and Claude-User. Blocking ClaudeBot only keeps your future content out of training data. Models may still know about you from earlier crawls, Common Crawl, and other licensed sources.
How often does ClaudeBot crawl?
Anthropic does not publish a fixed schedule. In practice, sites see bursts around model-training cycles rather than a steady daily rate. Use Crawl-delay in robots.txt if request volume is a concern.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- First-party measurement without third-party cookies
- Recognized AI referrals joined to Stripe payments
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
