Key point. Our AI crawler registry is one JSON file with 26 AI crawler and control tokens: vendor, purpose, robots.txt behaviour, documentation link and published IP-range link for each. Each row carries the date its vendor page was fetched: October 14 or October 18, 2026. For what each crawler does and the robots.txt lines to block or allow it, see AI crawlers.
What is in the file?
One object per token, with these fields:
token,vendor,product: the robots.txt name and who runs it.purposeandgroup: AI search, user fetch, training, web search or other. The file defines each value.kind: a crawler that visits (user_agent), a robots.txt name that never visits (control_token), or a legacy name.honours_robotsand its source:yes,may-not(the vendor says robots.txt may not apply when a person triggers the request),noorunclear.docs_url,ip_ranges_url,date_verified: the vendor page, its published IP list where there is one, and the day we found the token on that page.
A script builds the file from the list our AI crawler check uses, plus the agents our crawler-access study added. A test fails if the file and the checker ever disagree.
FAQ
Where can I read about each crawler?
On AI crawlers: every name in this file, filtered by purpose, with a page per crawler covering what it does, whether it follows robots.txt and the exact lines to block or allow it.
How often is the file updated?
Each row carries the date its vendor page was last checked. We re-check the pages and rebuild the file when vendors change their documentation.
Can I use the JSON in my own tools?
Yes, including in commercial tools. The data is licensed under CC BY 4.0: credit Elementera and link to this page. The file holds data only, no code.
Work with us
Want it done for you? Book a call about managed AEO services. Want to see where you stand first? Get your free AI visibility report from Lenz.