AI Crawler Directory · Apple
Applebot-Extended:
what it is and how to use it.
OperatorApple
PurposeAI training control token
Respects robots.txtYes
Executes JavaScriptNo
Hits verifiableYes — published IPs
Typerobots.txt control token
The token is itself a robots.txt control — that is its entire mechanism.
Applebot-Extended is Apple’s AI-training opt-out token — the Google-Extended pattern applied to Apple’s stack. It never crawls: pages are fetched by the regular Applebot (which also powers Siri and Spotlight suggestions), and the Applebot-Extended robots.txt token controls whether that crawled content may be used to train Apple’s foundation models, including those behind Apple Intelligence.
The split lets you stay in Siri and Spotlight results while opting out of model training: disallow Applebot-Extended, allow Applebot. Blocking Applebot itself, by contrast, removes you from Apple’s search surfaces entirely.
Because Applebot-Extended sends no requests, there is nothing to verify under its name. Authenticate the underlying Applebot traffic instead — Apple documents reverse-DNS verification against *.applebot.apple.com.
Applebot-Extended User-Agent string
No dedicated User-Agent — pages are fetched by Applebot (e.g. "Mozilla/5.0 … (Applebot/0.1; +http://www.apple.com/go/applebot)"); Applebot-Extended is a robots.txt token only.How to verify Applebot-Extended hits are real
The User-Agent header is plain text — any client can claim to be Applebot-Extended, and scrapers routinely do. Verify the underlying Applebot hits via reverse DNS to *.applebot.apple.com; Applebot-Extended itself never sends requests.
Authentic hits reverse-resolve to a hostname ending in .applebot.apple.com that forward-resolves back to the same IP:
host <hit-ip> # then: host <returned-hostname>Allow or block Applebot-Extended in robots.txt
This is a training-data control: allowing it lets your content shape what future models know; blocking it keeps your material out of training corpora without affecting live citations. A policy call, not a traffic one.
Allow
User-agent: Applebot-Extended
Allow: /Block
User-agent: Applebot-Extended
Disallow: /Applebot-Extended — frequently asked questions
What is Applebot-Extended?
Applebot-Extended is a robots.txt token that controls whether content crawled by Applebot may be used to train Apple’s foundation models (the models behind Apple Intelligence). It is a policy control, not a separate crawler.
Does blocking Applebot-Extended remove me from Siri or Spotlight?
No. Siri and Spotlight visibility comes from Applebot. Disallowing only Applebot-Extended keeps you in Apple’s search surfaces while excluding your content from Apple’s AI training.
Why does Applebot-Extended never appear in my logs?
Because it sends no requests — Applebot does the fetching, and Apple applies the Applebot-Extended rule as a data-use filter afterward. A log line claiming this UA is spoofed by definition.
How do I verify Applebot traffic?
Apple documents reverse-DNS verification: an authentic hit’s IP resolves to a hostname under applebot.apple.com, and that hostname resolves back to the same IP.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- Cookieless, so no consent banner for analytics
- Every AI referral matched to the payment it produced
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
