AI Crawler Directory · Anthropic
Claude-SearchBot:
what it is and how to verify it.
OperatorAnthropic
PurposeAI search indexing
Respects robots.txtYes
Executes JavaScriptNo
Hits verifiableYes — published IPs
TypeHTTP crawler / fetcher
Claude-SearchBot navigates the web to improve the quality and relevance of Claude’s search results. It plays the OAI-SearchBot role in Anthropic’s stack: building the index that Claude’s web-search features draw citations from.
Anthropic notes that disallowing Claude-SearchBot reduces your visibility in Claude’s search results — the direct trade. For sites competing on AI visibility, this bot is the gatekeeper for the Claude citation channel, so the standard posture is to allow it explicitly.
It respects robots.txt and does not execute JavaScript, which makes server-rendered content with clear structure the reliable way to be indexed completely.
Claude-SearchBot User-Agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +claude-searchbot@anthropic.com)Representative format — Anthropic does not publish an exact canonical UA string; the bot token is the stable identifier.
How to verify Claude-SearchBot hits are real
The User-Agent header is plain text — any client can claim to be Claude-SearchBot, and scrapers routinely do. Verify against Anthropic’s published bot IPs at claude.com/crawling/bots.json.
https://claude.com/crawling/bots.jsonFetch the list and check the hit’s source IP against it — for a quick manual look:
curl -s https://claude.com/crawling/bots.jsonAllow or block Claude-SearchBot in robots.txt
This is a search-index bot: it feeds the citations AI answers link to. If you want AI visibility — and the referral traffic behind it — the standard posture is to allow it.
Allow
User-agent: Claude-SearchBot
Allow: /Block
User-agent: Claude-SearchBot
Disallow: /Claude-SearchBot — frequently asked questions
What is Claude-SearchBot?
Claude-SearchBot is Anthropic’s search-indexing crawler. It crawls the web to improve search result quality for Claude users, powering which pages Claude’s search features can surface and cite.
Should I allow Claude-SearchBot?
Yes, if Claude citations matter to you. Anthropic explicitly documents that disabling it reduces your visibility in Claude search results. Blocking it is opting out of the Claude citation channel.
Is Claude-SearchBot used for model training?
No — training collection is ClaudeBot’s job. Anthropic separates the two, so you can run a split policy: block ClaudeBot to stay out of training data while allowing Claude-SearchBot to remain citable in live answers.
How do I verify Claude-SearchBot hits?
Check the source IP against claude.com/crawling/bots.json. Only an IP match proves the request came from Anthropic; the UA string alone can be forged by any client.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- Cookieless, so no consent banner for analytics
- Every AI referral matched to the payment it produced
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
