Key point. Diffbot-User is Diffbot's AI user fetcher. It fetches a page when a person asks the assistant to look at it; it is not an automatic crawler. It follows robots.txt.
Diffbot-User at a glance
These are the facts our registry holds for Diffbot-User, checked against Diffbot's documentation on October 22, 2026.
| Vendor | Diffbot |
| Used by | Diffbot Knowledge Graph and web search |
| Purpose | AI user fetch |
| Type | User agent |
| Follows robots.txt | Yes; source: community ai.robots.txt list (commit 9ad8a47e) |
| Vendor documentation | Diffbot crawler page, checked October 22, 2026 |
| Published IP ranges | None listed in our registry |
What is Diffbot-User?
Diffbot-User is Diffbot's AI user fetcher. It fetches a page when a person asks the assistant to look at it; it is not an automatic crawler.
Diffbot uses other names for other jobs: Diffbot (AI search). Each takes its own robots.txt group.
Does Diffbot-User follow robots.txt?
Yes, Diffbot-User follows robots.txt: our registry records this from the community ai.robots.txt list (commit 9ad8a47e).
A robots.txt rule records your request. It does not prove a crawler honours it, and firewall or CDN bot settings can refuse a crawler that robots.txt allows, so check those too.
How do I block Diffbot-User in robots.txt?
Add a group that names Diffbot-User and disallows the whole site:
User-agent: Diffbot-User
Disallow: /To keep it out of one folder only, disallow that path instead, for example Disallow: /private/.
How do I allow Diffbot-User?
Diffbot-User is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. A crawler obeys the most specific group that names it and falls back to User-agent: * only when no group names it (RFC 9309), so if a catch-all rule blocks crawlers, add a group for it:
User-agent: Diffbot-User
Allow: /How can I tell if a request really comes from Diffbot-User?
Treat the name alone as a claim. User agents can be faked, and our registry lists no published IP-range file for Diffbot-User, so there is no list to match the request's address against.
How often do startup websites block Diffbot-User?
We have no figure for Diffbot-User: it was not one of the crawlers our AI Crawler Access Report tracked. For the crawlers it did track, the headline figure is: “3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) block at least one AI search crawler (Elementera AI Crawler Access Report, October 2026).”
The method and the per-crawler table are in the AI Crawler Access Report.
How do I check whether my robots.txt blocks Diffbot-User?
Paste your robots.txt into our free checker and it shows, crawler by crawler, whether Diffbot-User may read the path you choose. Nothing you paste leaves the page.