AI Crawler Directory · Meta
meta-externalfetcher:
what it is and how to verify it.
OperatorMeta
PurposeUser-triggered fetches
Respects robots.txtNo (user-triggered)
Executes JavaScriptNo
Hits verifiableNo — nothing published
TypeHTTP crawler / fetcher
Meta documents it may bypass robots.txt because fetches are performed at a user’s request.
meta-externalfetcher is Meta’s on-demand fetcher. It retrieves individual links when a user request requires it, supporting product functions like Meta AI answering with live page content and the evaluation of agentic AI capabilities. It fetches specific pages rather than systematically indexing.
Because each request originates from a user action, Meta documents that it may bypass robots.txt — the same user-agency argument OpenAI and Perplexity make for their fetchers. The only enforceable block is at the network layer, and Meta publishes no IP list, which makes reliable filtering genuinely hard.
In attribution terms it plays the ChatGPT-User role for the Meta AI channel: hits indicate live human demand routed through Meta’s assistant surfaces.
meta-externalfetcher User-Agent string
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)How to verify meta-externalfetcher hits are real
The User-Agent header is plain text — any client can claim to be meta-externalfetcher, and scrapers routinely do. No published IP ranges — hits cannot be verified beyond the UA string.
Because nothing is published, the honest label for every meta-externalfetcher log line is unverified — it may be Meta, or a scraper borrowing the name.
Allow or block meta-externalfetcher in robots.txt
This is a user-triggered fetcher: each request is a live human asking about your page. Blocking it hides your content from users at the exact moment they want it.
Allow
User-agent: meta-externalfetcher
Allow: /Block
User-agent: meta-externalfetcher
Disallow: /
# Note: may bypass robots.txt for user-requested fetches.meta-externalfetcher — frequently asked questions
What is meta-externalfetcher?
meta-externalfetcher is Meta’s user-triggered link fetcher. It retrieves individual pages on demand to support features like Meta AI answers and agentic capabilities, rather than crawling sites systematically.
Does meta-externalfetcher respect robots.txt?
Not necessarily. Meta documents that it may bypass robots.txt because the fetch was requested by a user. Robots rules are advisory for this bot.
How is it different from meta-externalagent?
meta-externalagent crawls systematically for AI training and indexing; meta-externalfetcher fetches single links at a user’s request. Training corpus versus live user session — the same split OpenAI makes between GPTBot and ChatGPT-User.
Can I verify meta-externalfetcher hits?
No — Meta publishes no IP ranges for it. Any log entry with this UA should be treated as unverified; it may be Meta, or it may be a scraper borrowing the name.
See which AI crawlers hit your site — and what the traffic earns
Attrifast tracks AI referrals as revenue lines: ChatGPT, Perplexity, Claude, and Gemini visits tied to real Stripe money. $9.99/mo flat.
- One script tag and a Stripe key — live in minutes
- Cookieless, so no consent banner for analytics
- Every AI referral matched to the payment it produced
7-day free trial · $0 due today · then $9.99/mo · cancel anytime
