AI Crawler Directory · Google
Google-Extended:
what it is and how to use it.
OperatorGoogle
PurposeAI training control token
Respects robots.txtYes
Executes JavaScriptNo
Hits verifiableYes — published IPs
Typerobots.txt control token
The token itself is a robots.txt control — honoring it is its entire function.
Google-Extended is not a crawler — it is a robots.txt control token. You will never see "Google-Extended" in your server logs, because crawling is done by Google’s existing user agents. The token controls whether content Google has crawled may be used to train future generations of Gemini models and for grounding in Gemini Apps and the Vertex AI API.
Google documents the critical boundary: Google-Extended does not affect your inclusion or ranking in Google Search. Disallowing it keeps your content out of Gemini training and grounding without touching your search traffic — which makes it the cleanest AI-training opt-out any major vendor offers.
Note what it does not cover: AI Overviews in Google Search are part of Search itself, so the only way to stay out of them entirely is to stay out of Google Search — Google-Extended has no effect there.
Google-Extended User-Agent string
No dedicated User-Agent — crawling happens under existing Google UA strings (Googlebot and others); Google-Extended exists only as a robots.txt token.How to verify Google-Extended hits are real
The User-Agent header is plain text — any client can claim to be Google-Extended, and scrapers routinely do. There are no Google-Extended hits to verify — requests arrive as Googlebot/GoogleOther; verify those against Google’s published ranges.
https://developers.google.com/static/crawling/ipranges/common-crawlers.jsonFetch the list and check the hit’s source IP against it — for a quick manual look:
curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.jsonAllow or block Google-Extended in robots.txt
This is a training-data control: allowing it lets your content shape what future models know; blocking it keeps your material out of training corpora without affecting live citations. A policy call, not a traffic one.
Allow
User-agent: Google-Extended
Allow: /Block
User-agent: Google-Extended
Disallow: /Google-Extended — frequently asked questions
What is Google-Extended?
Google-Extended is a robots.txt token that controls whether your content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It is not a separate crawler — Google crawls with its normal user agents and applies the token as a policy filter.
Does blocking Google-Extended hurt my Google rankings?
No. Google explicitly documents that Google-Extended does not impact a site’s inclusion in Google Search and is not a ranking signal. It only controls AI training and grounding use.
Why does Google-Extended never appear in my logs?
Because it sends no requests. Crawling happens under existing Google UA strings; Google-Extended is consulted as a robots.txt rule when Google decides how crawled content may be used. Any log line claiming to be "Google-Extended" is a spoof by definition.
Does Google-Extended control AI Overviews?
No. AI Overviews are a Google Search feature, governed by normal Search indexing controls (Googlebot, noindex, nosnippet). Google-Extended governs Gemini model training and Gemini/Vertex grounding only.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- First-party measurement without third-party cookies
- Recognized AI referrals joined to Stripe payments
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
