Key point. Meta-ExternalFetcher is Meta's AI user fetcher. It fetches a page when a person asks the assistant to look at it; it is not an automatic crawler. robots.txt may not apply to it, because a person triggers the request.
Meta-ExternalFetcher at a glance
These are the facts our registry holds for Meta-ExternalFetcher, checked against Meta's documentation on October 14, 2026.
| Vendor | Meta |
| Purpose | AI user fetch |
| Type | User agent |
| Follows robots.txt | Not always; source: vendor's own crawler page, read October 14, 2026 |
| Vendor documentation | Meta crawler page, checked October 14, 2026 |
| Published IP ranges | None listed in our registry |
What is Meta-ExternalFetcher?
Meta-ExternalFetcher is Meta's AI user fetcher. It fetches a page when a person asks the assistant to look at it; it is not an automatic crawler. Our registry notes: Meta says Meta-ExternalFetcher fetches links at a user's request, so it may bypass robots.txt.
Meta uses other names for other jobs: Meta-ExternalAgent (AI training). Each takes its own robots.txt group.
Does Meta-ExternalFetcher follow robots.txt?
Not always, because a person triggers each request: Our registry notes: Meta says Meta-ExternalFetcher fetches links at a user's request, so it may bypass robots.txt.
What Meta says (Meta's crawler documentation, read October 14, 2026):
- “The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user.”
A robots.txt rule records your request. It does not prove a crawler honours it, and firewall or CDN bot settings can refuse a crawler that robots.txt allows, so check those too.
How do I block Meta-ExternalFetcher in robots.txt?
Add a group that names Meta-ExternalFetcher and disallows the whole site:
User-agent: Meta-ExternalFetcher
Disallow: /To keep it out of one folder only, disallow that path instead, for example Disallow: /private/.
robots.txt may not stop it. robots.txt may not apply to it, because a person triggers the request. If you must keep it out, the block has to happen in your firewall or CDN, not only in robots.txt.
How do I allow Meta-ExternalFetcher?
Meta-ExternalFetcher is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. A crawler obeys the most specific group that names it and falls back to User-agent: * only when no group names it (RFC 9309), so if a catch-all rule blocks crawlers, add a group for it:
User-agent: Meta-ExternalFetcher
Allow: /How can I tell if a request really comes from Meta-ExternalFetcher?
Treat the name alone as a claim. User agents can be faked, and our registry lists no published IP-range file for Meta-ExternalFetcher, so there is no list to match the request's address against.
How often do startup websites block Meta-ExternalFetcher?
Our AI Crawler Access Report read the robots.txt files of 600 sampled startup websites on October 7, 2026. This is its row for Meta-ExternalFetcher, across the 569 sites whose answer it could read:
| Crawler | Type | Blocked on at least one path | Blocked on all six paths | Named in the file |
|---|---|---|---|---|
| Meta-ExternalFetcher | AI user fetch | 4 of 569 sites (0.7%; 95% interval 0.3% to 1.8%) | 1 of 569 sites (0.2%; 95% interval 0.0% to 1.0%) | 8 of 569 sites (1.4%; 95% interval 0.7% to 2.8%) |
“Blocked” means the file disallows the crawler on at least one of six common paths (/, /pricing, /blog/, /docs/, /about, /product).
The report's headline figure, for the three AI search crawlers it tracked: “3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) block at least one AI search crawler (Elementera AI Crawler Access Report, October 2026).”
The figures describe that sample of Y Combinator-backed software companies on the day we fetched it. The method, the intervals and the full table are in the AI Crawler Access Report.
What the AI Crawler Access Report adds about Meta-ExternalFetcher:
5 of 569 sites (0.9%; 95% interval 0.4% to 2.0%) block at least one of the fetchers that visit a page when a person asks an assistant to read it (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher).
How do I check whether my robots.txt blocks Meta-ExternalFetcher?
Read the group by hand: our free crawler checker covers 13 crawlers, and Meta-ExternalFetcher is not one of them yet.
- Open
yoursite.com/robots.txt. - Look for a
User-agent: Meta-ExternalFetcherline. If there is one, only that group applies to Meta-ExternalFetcher. - If there is none, the
User-agent: *group applies. - Within the group, the longest matching
AlloworDisallowrule for a path wins.
Where can I see every AI crawler?
Our list of AI crawlers shows all 26 names in our registry, filtered by purpose. The same data is a free JSON download in our AI crawler registry.
Frequently asked questions
Three questions site owners ask about this name, each answered in one paragraph.
Should I block Meta-ExternalFetcher?
Only if you do not want Meta's assistant to read your pages when a person asks it to. robots.txt may not apply to it, because a person triggers the request. A block is therefore a request, not a guarantee.
Does blocking Meta-ExternalFetcher affect other crawlers?
No. robots.txt groups are matched by name, so a group for Meta-ExternalFetcher applies only to Meta-ExternalFetcher. Meta's other names (Meta-ExternalAgent) need their own groups.
How do I see whether Meta-ExternalFetcher visits my site?
Search your server or CDN logs for Meta-ExternalFetcher in the user agent. User agents can be faked, and our registry lists no published IP-range file for it, so treat the name alone as a claim.
Work with us
There are two ways to work with us. Teams that want the work done for them can use our managed AEO services: Book a call. Teams that want to see where they stand first can start with Lenz, our free analysis of how ChatGPT and Gemini answer their buyers' questions: Get your free AI visibility report.
How this page was made
A script builds this page from our AI crawler registry, the vendor crawler pages we read on October 14, 2026, and, for the blocking figures, the AI Crawler Access Report. Every sentence in quotation marks was checked against the vendor's page that day. Soroush Mehri reviewed the page, and Mojtaba Ebadi approved it for publication.
Sources
- Meta, crawler documentation: developers.facebook.com (read October 14, 2026)
- ai.robots.txt, community list of AI agents: github.com/ai-robots-txt/ai.robots.txt
- Koster, Illyes, Zeller and Sassman, “Robots Exclusion Protocol”, RFC 9309, IETF, 2022: rfc-editor.org
- Elementera, AI Crawler Access Report, October 2026: /resources/ai-crawler-access-report