Key point. Diffbot is Diffbot's AI search crawler for Diffbot Knowledge Graph and web search. It crawls pages so Diffbot Knowledge Graph and web search can find and cite them. robots.txt may not apply to it in every case, by its vendor's own documentation.
Diffbot at a glance
These are the facts our registry holds for Diffbot, checked against Diffbot's documentation on October 22, 2026.
| Vendor | Diffbot |
| Used by | Diffbot Knowledge Graph and web search |
| Purpose | AI search |
| Type | User agent |
| Follows robots.txt | Not always; source: vendor's documentation, as recorded in our AI crawler check |
| Vendor documentation | Diffbot crawler page, checked October 22, 2026 |
| Published IP ranges | None listed in our registry |
What is Diffbot?
Diffbot is Diffbot's AI search crawler for Diffbot Knowledge Graph and web search. It crawls pages so Diffbot Knowledge Graph and web search can find and cite them. Our registry notes: Diffbot says its crawler is not used for AI training, and that robots.txt can be overridden where Diffbot has an agreement with the site.
Diffbot uses other names for other jobs: Diffbot-User (AI user fetch). Each takes its own robots.txt group.
Does Diffbot follow robots.txt?
Not always, because a person triggers each request: Our registry notes: Diffbot says its crawler is not used for AI training, and that robots.txt can be overridden where Diffbot has an agreement with the site.
A robots.txt rule records your request. It does not prove a crawler honours it, and firewall or CDN bot settings can refuse a crawler that robots.txt allows, so check those too.
How do I block Diffbot in robots.txt?
Add a group that names Diffbot and disallows the whole site:
User-agent: Diffbot
Disallow: /To keep it out of one folder only, disallow that path instead, for example Disallow: /private/.
robots.txt may not stop it. robots.txt may not apply to it in every case, by its vendor's own documentation. If you must keep it out, the block has to happen in your firewall or CDN, not only in robots.txt.
How do I allow Diffbot?
Diffbot is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. A crawler obeys the most specific group that names it and falls back to User-agent: * only when no group names it (RFC 9309), so if a catch-all rule blocks crawlers, add a group for it:
User-agent: Diffbot
Allow: /Want the exact block for every AI crawler at once? Our robots.txt generator for AI crawlers writes it for each choice.
How can I tell if a request really comes from Diffbot?
Treat the name alone as a claim. User agents can be faked, and our registry lists no published IP-range file for Diffbot, so there is no list to match the request's address against.
How often do startup websites block Diffbot?
We have no figure for Diffbot: it was not one of the crawlers our AI Crawler Access Report tracked. For the crawlers it did track, the headline figure is: “3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) block at least one AI search crawler (Elementera AI Crawler Access Report, October 2026).”
The method and the per-crawler table are in the AI Crawler Access Report.
How do I check whether my robots.txt blocks Diffbot?
Paste your robots.txt into our free checker and it shows, crawler by crawler, whether Diffbot may read the path you choose. Nothing you paste leaves the page.