AI Crawler Directory
Every AI crawler that visits your site —
and how to verify them.
17 AI bots, documented from official sources: what each one does, whether it respects robots.txt, and how to prove a hit really came from OpenAI, Anthropic, or Perplexity — not a scraper borrowing the name.
BotOperatorPurposeVerifiable?
GPTBotOpenAIAI model trainingYesChatGPT-UserOpenAIUser-triggered fetchesYesOAI-SearchBotOpenAIAI search indexingYesClaudeBotAnthropicAI model trainingYesClaude-UserAnthropicUser-triggered fetchesYesClaude-SearchBotAnthropicAI search indexingYesPerplexityBotPerplexityAI search indexingYesPerplexity-UserPerplexityUser-triggered fetchesYesGoogle-ExtendedGoogleAI training control tokenYesGoogleOtherGoogleR&D and product crawlingYesCCBotCommon CrawlOpen web corpus (training feedstock)Yesmeta-externalagentMetaAI model trainingNometa-externalfetcherMetaUser-triggered fetchesNoBytespiderByteDanceAI model trainingNoAmazonbotAmazonAlexa answers + AI improvementYesApplebot-ExtendedAppleAI training control tokenYesDuckAssistBotDuckDuckGoAI-assisted answersYesFacts from each vendor’s official crawler documentation, as of July 2026. Click any bot for the full profile, robots.txt snippets, and verification steps.
Three kinds of AI bot — three different stakes
“AI crawler” lumps together bots with completely different value to you. The purpose column above is the one that matters.
AI search indexing
OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot. These build the indexes AI answers cite from. Blocking them is opting out of AI search the way blocking Googlebot opts you out of Google. If you want AI citations — and the referral revenue behind them — these stay allowed.
User-triggered fetches
ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher. Each hit is a live human reading your page through an AI assistant right now — the strongest demand signal in your logs, one step before a referral visit. Most ignore robots.txt because the user, not the bot, initiated the request.
AI model training
GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent. Corpus collection for future models — no live reader behind any hit. Allowing them shapes what AI assistants "know" about you natively; blocking them is a clean data-policy call that costs no citations today.
Why the User-Agent string proves nothing
Any scraper can send GPTBot as its User-Agent — and many do, to borrow OpenAI’s reputation and slip past bot rules. That is why OpenAI, Anthropic, Perplexity, DuckDuckGo, Amazon, and Common Crawl publish the exact IP ranges their bots crawl from. A hit is verified only when its source IP falls inside a published range; everything else is a claim. Bots with no published ranges — Bytespider, Meta’s crawlers — can never be authenticated, which is itself useful to know when you read your logs.
Each bot page below lists the official IP endpoint and a one-line check you can run. Or skip the manual work: run your site through the free AI readiness checker to see which bots your robots.txt currently allows.
Knowing which bots you allow is a snapshot; knowing which ones actually showed up is a log problem. Attrifast records every hit server-side and checks the source IP against each vendor’s published ranges — the setup is documented under AI crawler tracking.
AI crawlers — frequently asked questions
Should I block AI crawlers?
It depends on the bot’s job. Search and user-fetch bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User) are the pipeline to AI citations and referral traffic — blocking them removes you from AI answers. Training bots (GPTBot, ClaudeBot, CCBot, Bytespider) are a policy call: allowing them puts your content in future models; blocking them keeps it out without affecting live citations.
How do I know if an AI crawler hit is real?
Never trust the User-Agent string — anyone can send "GPTBot" with curl. OpenAI, Anthropic, Perplexity, DuckDuckGo, Amazon, and Common Crawl publish the IP ranges their bots crawl from; a hit is verified only when its source IP falls inside a published range. Bots without published ranges (Bytespider, Meta’s crawlers) can never be verified.
Which AI crawlers send traffic back to my site?
Search-index bots and user-fetch bots precede human visits: OAI-SearchBot indexes pages ChatGPT search cites, ChatGPT-User and Perplexity-User fetch pages live users are asking about, and DuckAssistBot powers source-cited answers. Training crawlers (GPTBot, ClaudeBot, CCBot) do not correspond to any live reader.
Do AI crawlers show up in Google Analytics?
No. AI crawlers do not execute JavaScript, so tag-based analytics never sees them. They only appear in server logs, CDN logs, or a server-side crawler tracker. That is why sites often think no AI bot has ever visited when their logs show hundreds of hits.
What is the difference between training, search, and user-fetch bots?
Training bots collect corpus data for future models on their own schedule. Search bots build the index that AI answers cite from. User-fetch bots retrieve a page in real time because a human just asked about it — they are the strongest demand signal, one step before a referral visit.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- Cookieless, so no consent banner for analytics
- Every AI referral matched to the payment it produced
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
