AI Crawler Directory · Google
GoogleOther is Google’s generic crawler, used by various product teams to fetch publicly accessible content — including one-off crawls for internal research and development. Google introduced it to take non-Search workloads off Googlebot, so that a Googlebot hit means Search indexing and nothing else.
For site owners the practical meaning is ambiguity: a GoogleOther hit tells you a Google team wanted your content, but not which team or why. AI-related R&D crawling is widely assumed to be part of its workload, which is why GoogleOther shows up in AI-crawler audits despite its generic charter.
It shares infrastructure with Googlebot — same published IP ranges, same crawl-*.googlebot.com reverse DNS — and it respects robots.txt, so you can rate-limit or block it without touching Googlebot’s Search crawling.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther)The User-Agent header is plain text — any client can claim to be GoogleOther, and scrapers routinely do. GoogleOther crawls from Googlebot’s published ranges; reverse DNS resolves to crawl-*.googlebot.com.
https://developers.google.com/static/crawling/ipranges/common-crawlers.jsonFetch the list and check the hit’s source IP against it — for a quick manual look:
curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.jsonThis is a general product crawler: blocking it does not affect search rankings or AI citations, so throttle or block freely if crawl volume bothers you.
User-agent: GoogleOther
Allow: /User-agent: GoogleOther
Disallow: /GoogleOther is a generic Google crawler used by product teams for fetching public content, such as one-off crawls for internal research and development. It is separate from Googlebot, which handles Search indexing.
No. Search indexing belongs to Googlebot. GoogleOther exists precisely so non-Search crawling can be controlled independently — blocking it does not affect rankings or indexing.
The same way as Googlebot: check the IP against Google’s published crawler ranges (common-crawlers.json) or confirm reverse DNS resolves to crawl-*.googlebot.com and forward-resolves back to the same IP.
Google describes it only as a generic product/R&D crawler. AI training use of already-crawled content is governed by the Google-Extended robots token, not by which bot fetched the page.
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
