AI Crawler Directory · OpenAI
GPTBot:
what it is and how to verify it.
OperatorOpenAI
PurposeAI model training
Respects robots.txtYes
Executes JavaScriptNo
Hits verifiableYes — published IPs
TypeHTTP crawler / fetcher
GPTBot is OpenAI’s training crawler. It systematically fetches publicly accessible web pages to build the corpus that future GPT foundation models are trained on. It is not the bot that answers a live ChatGPT question — that job belongs to ChatGPT-User and OAI-SearchBot — so a GPTBot hit never corresponds to a human reading your page in real time.
GPTBot visits on its own schedule, the way a search-engine spider does: it discovers URLs, fetches HTML, and moves on. It does not execute JavaScript, so anything rendered only on the client is invisible to it. Sites that server-render their content — including their pricing, product facts, and documentation — give the model complete information to learn from.
Whether to allow GPTBot is a policy decision, not a traffic decision. Allowing it means your content can inform what future GPT models "know"; blocking it keeps your material out of training data but has no effect on ChatGPT search citations today, which are controlled separately by OAI-SearchBot.
GPTBot User-Agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotHow to verify GPTBot hits are real
The User-Agent header is plain text — any client can claim to be GPTBot, and scrapers routinely do. OpenAI publishes the exact IP ranges GPTBot crawls from as machine-readable JSON.
https://openai.com/gptbot.jsonFetch the list and check the hit’s source IP against it — for a quick manual look:
curl -s https://openai.com/gptbot.jsonAllow or block GPTBot in robots.txt
This is a training-data control: allowing it lets your content shape what future models know; blocking it keeps your material out of training corpora without affecting live citations. A policy call, not a traffic one.
Allow
User-agent: GPTBot
Allow: /Block
User-agent: GPTBot
Disallow: /GPTBot — frequently asked questions
What is GPTBot?
GPTBot is OpenAI’s web crawler for collecting publicly available content that may be used to train its generative AI foundation models. It identifies itself with the token GPTBot in the User-Agent header (GPTBot/1.4 as of July 2026) and respects robots.txt rules.
Does GPTBot respect robots.txt?
Yes. OpenAI documents that GPTBot honors robots.txt allow and disallow directives. Add "User-agent: GPTBot / Disallow: /" to block it, or scope the disallow to specific paths. Changes take effect the next time the bot refetches your robots.txt.
Does a GPTBot visit mean I will get traffic from ChatGPT?
Not directly. GPTBot collects training data; it does not fetch pages for live answers. Citation-driven visits come via OAI-SearchBot (search index) and ChatGPT-User (on-demand fetches). Blocking or allowing GPTBot does not change whether ChatGPT search cites you today.
How do I verify a GPTBot hit is really from OpenAI?
Check the source IP against OpenAI’s published ranges at openai.com/gptbot.json. The User-Agent string alone proves nothing — anyone can send "GPTBot" with curl. A hit is authentic only when its IP falls inside a published range.
Should I block GPTBot?
If your content is your product (paywalled journalism, licensed data), blocking is reasonable. If you sell software or services and want AI assistants to know you exist, most sites gain more from being in the training corpus than they lose. It is a policy call — there is no ranking penalty either way.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- Cookieless, so no consent banner for analytics
- Every AI referral matched to the payment it produced
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
