AI Crawler Directory · OpenAI
OAI-SearchBot:
what it is and how to verify it.
OperatorOpenAI
PurposeAI search indexing
Respects robots.txtYes
Executes JavaScriptNo
Hits verifiableYes — published IPs
TypeHTTP crawler / fetcher
OAI-SearchBot builds and refreshes the index behind ChatGPT’s search features. When ChatGPT answers with linked, cited sources, those candidates come from what OAI-SearchBot has crawled. It is explicitly not used to train foundation models — OpenAI separates search indexing (this bot) from training collection (GPTBot).
For any site that wants ChatGPT citations — and the referral traffic they drive — this is the OpenAI bot that matters most. If OAI-SearchBot cannot reach your pages, ChatGPT search cannot cite them, regardless of how good the content is. Blocking it removes you from ChatGPT’s search surface the same way blocking Googlebot removes you from Google.
It crawls politely and respects robots.txt. Since it does not execute JavaScript, server-rendered pages with clean HTML, descriptive titles, and structured data get indexed most completely.
OAI-SearchBot User-Agent string
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbotHow to verify OAI-SearchBot hits are real
The User-Agent header is plain text — any client can claim to be OAI-SearchBot, and scrapers routinely do. OpenAI publishes OAI-SearchBot’s crawl IP ranges at openai.com/searchbot.json.
https://openai.com/searchbot.jsonFetch the list and check the hit’s source IP against it — for a quick manual look:
curl -s https://openai.com/searchbot.jsonAllow or block OAI-SearchBot in robots.txt
This is a search-index bot: it feeds the citations AI answers link to. If you want AI visibility — and the referral traffic behind it — the standard posture is to allow it.
Allow
User-agent: OAI-SearchBot
Allow: /Block
User-agent: OAI-SearchBot
Disallow: /OAI-SearchBot — frequently asked questions
What is OAI-SearchBot?
OAI-SearchBot is OpenAI’s search crawler. It indexes websites so they can be surfaced and cited in ChatGPT’s search features. OpenAI documents that it is not used to collect training data for foundation models.
Should I allow OAI-SearchBot?
Yes, if you want ChatGPT to cite you. It is the gateway to ChatGPT search visibility — blocking it is equivalent to opting out of a search engine. Sites competing for AI citations should explicitly allow it in robots.txt.
Does OAI-SearchBot train GPT models on my content?
No. OpenAI separates the two: GPTBot collects training data, OAI-SearchBot indexes for search. You can block GPTBot (no training) while allowing OAI-SearchBot (stay citable) — many publishers run exactly this split policy.
How do I verify OAI-SearchBot hits?
Compare the source IP against openai.com/searchbot.json. The published JSON is the source of truth; a matching User-Agent with a non-matching IP is a spoof.
Does an OAI-SearchBot crawl mean ChatGPT will cite me?
It is necessary but not sufficient — the same way a Googlebot crawl does not guarantee a ranking. Getting crawled puts you in the index; whether ChatGPT cites you depends on how well your content answers the prompts users actually ask.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- Cookieless, so no consent banner for analytics
- Every AI referral matched to the payment it produced
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
