AI Crawler Directory · Anthropic
Claude-User:
what it is and how to verify it.
OperatorAnthropic
PurposeUser-triggered fetches
Respects robots.txtPartially
Executes JavaScriptNo
Hits verifiableYes — published IPs
TypeHTTP crawler / fetcher
Fetches happen because a user asked; treat robots.txt as advisory for this bot.
Claude-User fetches web pages on behalf of a live Claude session. When someone asks Claude to look at a URL or pull current information as part of a conversation, this bot makes that request. Each hit corresponds to a real person consuming your content through Claude at that moment.
Like ChatGPT-User for OpenAI, this is the AI-traffic signal closest to a human visit: Claude summarizes or cites your page, and interested users click through to claude.ai-referred sessions. If you track AI attribution, Claude-User hits are the upstream event before Claude referral revenue appears.
Blocking it in robots.txt prevents Claude from retrieving your content for user-directed searches — which usually costs more visibility than it saves in server load, since request volume is naturally bounded by real user demand.
Claude-User User-Agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com)Representative format — Anthropic does not publish an exact canonical UA string; the bot token is the stable identifier.
How to verify Claude-User hits are real
The User-Agent header is plain text — any client can claim to be Claude-User, and scrapers routinely do. Authentic Claude-User hits come from the IPs Anthropic publishes at claude.com/crawling/bots.json.
https://claude.com/crawling/bots.jsonFetch the list and check the hit’s source IP against it — for a quick manual look:
curl -s https://claude.com/crawling/bots.jsonAllow or block Claude-User in robots.txt
This is a user-triggered fetcher: each request is a live human asking about your page. Blocking it hides your content from users at the exact moment they want it.
Allow
User-agent: Claude-User
Allow: /Block
User-agent: Claude-User
Disallow: /Claude-User — frequently asked questions
What is Claude-User?
Claude-User is Anthropic’s user-triggered fetcher. It accesses a website when an individual asks Claude a question that requires retrieving that site’s content — one fetch per user action, not systematic crawling.
Why do Claude-User hits matter?
Each one is a live human reading your content inside Claude. They are the precursor to claude.ai referral visits, so sites measuring AI revenue watch Claude-User activity as top-of-funnel demand for the Claude channel.
How do I verify Claude-User hits?
Match source IPs against claude.com/crawling/bots.json. Anthropic maintains one verified-IP endpoint for its bots, so ClaudeBot, Claude-User, and Claude-SearchBot can all be authenticated the same way.
What happens if I block Claude-User?
Anthropic states that disabling Claude-User prevents content retrieval when users direct Claude to your site. Claude will tell the user it could not access the page — the demand existed, and the answer came from a competitor’s page instead.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- First-party measurement without third-party cookies
- Recognized AI referrals joined to Stripe payments
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
