Key point. anthropic-ai is Anthropic's legacy robots.txt token. It is an older name: Anthropic documents other crawler names today. We found no clear statement on whether it follows robots.txt.
anthropic-ai at a glance
These are the facts our registry holds for anthropic-ai. No vendor page documents it.
| Vendor | Anthropic |
| Purpose | Other |
| Type | Legacy token |
| Follows robots.txt | No clear statement found; source: community ai.robots.txt list (commit 9ad8a47e) |
| Vendor documentation | None found: no vendor page documents this token |
| Published IP ranges | None listed in our registry |
What is anthropic-ai?
anthropic-ai is Anthropic's legacy robots.txt token. It is an older name: Anthropic documents other crawler names today. Anthropic's crawler help page, read on October 14, 2026, does not mention it; that page documents ClaudeBot, Claude-User and Claude-SearchBot.
Anthropic uses other names for other jobs: Claude-SearchBot (AI search); Claude-User (AI user fetch); ClaudeBot (AI training). Each takes its own robots.txt group.
Does anthropic-ai follow robots.txt?
We found no clear statement on whether anthropic-ai follows robots.txt; our registry records it as unclear, from the community ai.robots.txt list (commit 9ad8a47e).
A robots.txt rule records your request. It does not prove a crawler honours it, and firewall or CDN bot settings can refuse a crawler that robots.txt allows, so check those too.
How do I block anthropic-ai in robots.txt?
Add a group that names anthropic-ai and disallows the whole site:
User-agent: anthropic-ai
Disallow: /To keep it out of one folder only, disallow that path instead, for example Disallow: /private/.
How do I allow anthropic-ai?
anthropic-ai is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. A crawler obeys the most specific group that names it and falls back to User-agent: * only when no group names it (RFC 9309), so if a catch-all rule blocks crawlers, add a group for it:
User-agent: anthropic-ai
Allow: /How can I tell if a request really comes from anthropic-ai?
anthropic-ai is a robots.txt name, not a crawler on Anthropic's current crawler page, so there is nothing to verify. Check requests from Claude-SearchBot, Claude-User, ClaudeBot instead.
How often do startup websites block anthropic-ai?
We have no figure for anthropic-ai: it was not one of the crawlers our AI Crawler Access Report tracked. For the crawlers it did track, the headline figure is: “3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) block at least one AI search crawler (Elementera AI Crawler Access Report, October 2026).”
The method and the per-crawler table are in the AI Crawler Access Report.
What the AI Crawler Access Report adds about anthropic-ai:
ClaudeBot is Anthropic's training crawler; Claude-SearchBot indexes pages for search in Claude. Anthropic says disabling Claude-SearchBot may reduce a site's visibility and accuracy in user search results. A third name, Claude-User, fetches a page when a person asks Claude to read it. Each takes its own robots.txt group.
How do I check whether my robots.txt blocks anthropic-ai?
Read the group by hand: our free crawler checker covers 13 crawlers, and anthropic-ai is not one of them yet.
- Open
yoursite.com/robots.txt. - Look for a
User-agent: anthropic-ailine. If there is one, only that group applies to anthropic-ai. - If there is none, the
User-agent: *group applies. - Within the group, the longest matching
AlloworDisallowrule for a path wins.
Where can I see every AI crawler?
Our list of AI crawlers shows all 26 names in our registry, filtered by purpose. The same data is a free JSON download in our AI crawler registry.
Frequently asked questions
Three questions site owners ask about this name, each answered in one paragraph.
Should I block anthropic-ai?
It does no harm to keep, but it is not the name to rely on. Anthropic's crawler help page, read on October 14, 2026, does not mention it; that page documents ClaudeBot, Claude-User and Claude-SearchBot. To control Anthropic's training crawler, add a group for ClaudeBot.
Does blocking anthropic-ai affect other crawlers?
No. robots.txt groups are matched by name, so a group for anthropic-ai applies only to anthropic-ai. Anthropic's other names (Claude-SearchBot, Claude-User, ClaudeBot) need their own groups.
How do I see whether anthropic-ai visits my site?
Look for Anthropic's current names in your logs instead: Claude-SearchBot, Claude-User, ClaudeBot.
Work with us
There are two ways to work with us. Teams that want the work done for them can use our managed AEO services: Book a call. Teams that want to see where they stand first can start with Lenz, our free analysis of how ChatGPT and Gemini answer their buyers' questions: Get your free AI visibility report.
How this page was made
A script builds this page from our AI crawler registry, the vendor crawler pages we read on October 14, 2026, and, for the blocking figures, the AI Crawler Access Report. Every sentence in quotation marks was checked against the vendor's page that day. Soroush Mehri reviewed the page, and Mojtaba Ebadi approved it for publication.
Sources
- ai.robots.txt, community list of AI agents: github.com/ai-robots-txt/ai.robots.txt
- Koster, Illyes, Zeller and Sassman, “Robots Exclusion Protocol”, RFC 9309, IETF, 2022: rfc-editor.org
- Elementera, AI Crawler Access Report, October 2026: /resources/ai-crawler-access-report