Key point. Not supported for startup websites: of the 569 startup sites whose robots.txt we could read, 3 block a crawler AI search engines use. Training crawlers are blocked more often, and nearly always on purpose, so for most companies crawler access is not where AI visibility is lost.
Verdict: not supported for startup websites. Almost none block the crawlers that decide whether ChatGPT, Claude or Perplexity can find their pages, and most files do not mention AI crawlers at all.
What did we measure?
We read the robots.txt file of 600 randomly sampled websites of Y Combinator-backed B2B software, fintech and developer-tool companies, and checked, crawler by crawler, what each file allows.
- AI search crawlers are almost never blocked. 3 of 569 sites (0.5%; 95% interval 0.2% to 1.5%) block at least one of OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- Training crawlers are blocked more often, and on purpose. 18 of 569 sites (3.2%; 95% interval 2.0% to 4.9%) block at least one AI training crawler, and in 15 of those 18 the file names the crawler.
- Most files say nothing about AI at all. 467 of 569 sites (82%; 95% interval 79% to 85%) name no AI crawler, so every AI crawler is allowed by default.
A newer kind of line is rarer still: "26 of 569 sites (4.6%; 95% interval 3.1% to 6.6%) carry a Content-Signal line in robots.txt, which states preferences such as search, ai-input and ai-train, and 2 of 569 (0.4%; 95% interval 0.1% to 1.3%) say no to AI input."
Method
The table gives the sample, the intervals and the date behind each number above.
| Item | Detail |
|---|---|
| Sample | 600 randomly sampled websites of Y Combinator-backed B2B software, fintech and developer-tool companies; 569 readable |
| What we read | robots.txt only, main host only, crawler by crawler |
| 95% intervals | 0.2% to 1.5% (search); 2.0% to 4.9% (training); 79% to 85% (no AI crawler named); 3.1% to 6.6% and 0.1% to 1.3% (Content-Signal) |
| Date | October 7, 2026 |
| Design | Per site, descriptive. A robots.txt line states a preference; it does not show what a crawler reads or obeys |
Each number has a methodology card. The full report, the per-site dataset and a free checker are in the AI Crawler Access Report.
Where does the claim come from, and what do the docs say now?
The claim usually rests on counts of large news websites, and it mixes up two kinds of crawler that the vendors now document separately.
| Source | What it says | Published | Read |
|---|---|---|---|
| Research on news sites: Reuters Institute, How many news websites block AI crawlers? | By the end of 2023, 48% of the most widely used news websites across ten countries were blocking OpenAI's crawlers. | February 22, 2024 | October 22, 2026 |
| Current documentation: OpenAI, Overview of OpenAI crawlers | OAI-SearchBot (search) and GPTBot (training) are separate settings: a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. A site that blocks OAI-SearchBot is not shown in ChatGPT search answers. | current page | October 22, 2026 |
| The common claim | "Most companies block AI crawlers", so blocking is normal and costs nothing. | repeated widely | October 22, 2026 |
What the common claim gets wrong. A count of news publishers says nothing about company websites, and blocking GPTBot is a training decision, not a search one. The crawler that decides whether ChatGPT search can show your pages is OAI-SearchBot.
How far does the result reach?
The result covers startup websites: Y Combinator-backed B2B software, fintech and developer-tool companies. It does not describe news publishers, large retailers or any other kind of site, where blocking may be far more common.
It reads robots.txt only. A firewall or bot-protection setting can still block a crawler that robots.txt allows, so check both.
What should you do instead?
Check your own file and your firewall, then spend your effort further down the funnel: whether engines read, trust, describe and recommend you.
- Open your robots.txt and confirm that OAI-SearchBot, Claude-SearchBot and PerplexityBot are not blocked.
- Decide separately, as a training question, about GPTBot, ClaudeBot and Google-Extended.
- Ask whoever runs your firewall or CDN whether bot protection blocks those agents even where robots.txt allows them.
- Once access is clear, look at whether engines read, trust, describe and recommend you.
Work with us
If you want a team to measure and improve how AI search treats your company, our managed AEO services do that work with you: Book a call. If you would rather see where you stand first, Lenz gives you a free analysis of how ChatGPT and Gemini answer your buyers' questions: Get your free AI visibility report.
More claims we tested.