AI Crawler Access Report: the dataset
Key point. This page holds the data behind the AI Crawler Access Report: one row for each of the 600 sites we sampled. Each site has a random id instead of a name, so the table shows patterns, not companies.
How the sample was drawn, how the files were fetched and how each rule was read are described in the report's method section.
Download the full dataset
The full per-site file, one row for each of the 600 sites and one column for each of the 18 crawlers we tracked, is here: elementera-ai-crawler-access-report-2026-10.csv. It is a CSV file; each crawler has a state (allowed, blocked on some paths, blocked on all paths), the rule that decided it, and how we classified any block.
How to cite this dataset
You are welcome to use and quote this data. Please cite the report and link to it:
Elementera (2026). The AI Crawler Access Report: robots.txt rules for AI crawlers on 600 startup websites [Data set]. Data collected October 7, 2026. https://www.elementera.com/resources/ai-crawler-access-report-datasetData licence: CC BY 4.0: free to reuse, including commercially, with credit to Elementera and a link to this page.
Want to see how your own site would score? Paste your robots.txt into our free AI crawler checker; it reads the file with the same rules as this study.
How to read the tables
- robots.txt: "read" means the site returned a robots.txt file we could parse. "No file (4xx)" means it answered with a "not found" or other 4xx code, which under the robots.txt standard (RFC 9309) means no restrictions. Unreachable sites, and sites whose robots.txt redirected to another domain, are left out of every share in the report.
- Blocked: the file disallows the crawler on at least one of six common paths (
/,/pricing,/blog/,/docs/,/about,/product). A rule that only covers paths such as/admin/does not count. - AI search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. AI training crawlers or controls: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent and Bytespider. AI user fetchers: ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher.
Of the 569 sites whose answer we could read, 3 block at least one AI search crawler and 18 block at least one AI training crawler; the report gives each share with its 95% interval.
The sample by industry
| Industry | Sites sampled | robots.txt read | No file (4xx) | Unreachable or redirected | Blocks any AI crawler |
|---|---|---|---|---|---|
| Developer tools | 180 | 144 | 24 | 12 | 3 |
| Fintech | 107 | 86 | 17 | 4 | 5 |
| Other B2B | 313 | 266 | 32 | 15 | 10 |
Every site that blocks an AI crawler
18 of the 569 sites we could read block at least one AI crawler. All of them are listed here; every other site allows every AI crawler we tracked.
| Site id | Industry | YC batch | AI search crawlers blocked | AI training crawlers blocked | AI user fetchers blocked |
|---|---|---|---|---|---|
| S04289182 | Other B2B | 2025 or later | none | Meta-ExternalAgent, Bytespider | none |
| S1afb9699 | Other B2B | 2025 or later | none | GPTBot, ClaudeBot, CCBot, Applebot-Extended, Bytespider | none |
| S2fa08efc | Other B2B | 2023 to 2024 | none | CCBot, Bytespider | none |
| S38a2dab4 | Other B2B | 2020 to 2022 | none | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Bytespider | none |
| S3931d1a6 | Other B2B | 2023 to 2024 | none | CCBot, Bytespider | none |
| S394cd708 | Fintech | 2020 to 2022 | none | Meta-ExternalAgent, Bytespider | none |
| S3f770cf2 | Developer tools | 2025 or later | none | GPTBot, ClaudeBot, Google-Extended, CCBot | none |
| S432e6145 | Fintech | 2025 or later | none | Bytespider | Meta-ExternalFetcher |
| S44c14dac | Developer tools | 2023 to 2024 | OAI-SearchBot, Claude-SearchBot, PerplexityBot | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider | ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher |
| S5406634a | Fintech | 2020 to 2022 | none | GPTBot, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider | none |
| S6f6c6ea5 | Other B2B | 2025 or later | Claude-SearchBot | CCBot, Meta-ExternalAgent, Bytespider | Claude-User, Perplexity-User, Meta-ExternalFetcher |
| Sbbd3d3a5 | Other B2B | 2025 or later | none | Bytespider | none |
| Sdc0b08f0 | Developer tools | 2023 to 2024 | none | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider | none |
| Sdecbb7d0 | Fintech | 2019 or earlier | none | GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider | none |
| Se0a5d63a | Other B2B | 2020 to 2022 | OAI-SearchBot, Claude-SearchBot, PerplexityBot | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider | ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher |
| Se6343c40 | Other B2B | 2025 or later | none | GPTBot, CCBot | ChatGPT-User |
| Sf3ba6461 | Other B2B | 2019 or earlier | none | GPTBot | none |
| Sf803d0b0 | Fintech | 2025 or later | none | GPTBot, CCBot, Bytespider | none |
How this page was made
Our own scripts produced this table from the same data as the report, and they check that every count on this page matches the report before the page is written. Mojtaba Ebadi checked the page against the data, Soroush Mehri reviewed it, and Mojtaba Ebadi approved it for publication.
Sr. Engineering Leader with 15+ years of experience. Ex-ApplyBoard. AWS Certified Solutions Architect.
Ex-Microsoft. 10+ years of experience in AI/ML, MSc from Université de Montréal (Mila). US patent holder.
Related articles
- Dental Practice Software in 2026: A Practical Guide to PMS Platforms and AI Tools (US and Canada)
- Google Preferred Sources and AEO: what it is and how it affects your AI visibility
- What an AI Visibility Audit Actually Covers for SMBs
- Inside Google’s First Year of AI Mode: What 1 Billion Users Reveal About How Americans Search Now
- What You Might Have Missed From Google I/O 2026 (And Why It Matters for AEO and AI Search)