Free AI crawler checker: does your robots.txt let ChatGPT, Claude and Perplexity in?

AI answer engines can only cite pages their crawlers are allowed to fetch. Each vendor runs separate crawlers for search, for fetching a page a user asks about, and for model training, and one line in robots.txt can block the wrong one. This checker reads your file with the same rules we used to read 600 startup websites for the AI Crawler Access Report, and tells you in seconds which is which.

How does the checker work?

The checker parses your robots.txt in your browser and decides, for each AI crawler, whether it may fetch the path you choose. It follows the robots.txt standard (RFC 9309): a crawler obeys the most specific group that names it, falls back to User-agent: * only when no group names it, and within a group the longest matching rule wins.

1. Open your robots.txt. Go to https://your-site/robots.txt in a new tab and copy everything on the page.
2. Paste it into the box. Or try one of the three examples to see what a training-only opt-out, an accidental block and a file with no AI rules look like.
3. Choose the paths to check. The checker starts with /; add /pricing, /blog/ or /docs/ to check the pages buyers ask about.
4. Read the cards. Each vendor gets one card with three rows (search, user fetch, training) marked allowed or blocked, plus a summary of how many AI search engines may crawl the path.

The checker works on pasted text because browsers cannot fetch another site's robots.txt directly. That is also why nothing leaves your screen.

Which AI crawlers does it check?

It checks the crawlers of five vendors, OpenAI, Anthropic, Perplexity, Google and Microsoft, split by what each crawler does.

The crawler list was last checked against each vendor's own documentation on October 5, 2026, and the date is shown in the tool.

What do most websites allow?

Almost every website allows AI search crawlers. In our AI Crawler Access Report, 3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) blocked at least one AI search crawler, and 18 of 569 (3.2%; 95% interval 2.0% to 4.9%) blocked at least one AI training crawler, usually by naming it. If your check shows a blocked search crawler, you are in a very small group, and it is worth finding out why.

What should I do if a search crawler is blocked?

Fix the rule that blocks it, then check the layer robots.txt cannot show. A blocked search crawler can come from a catch-all User-agent: * rule written for something else.

1. Give each AI search crawler its own group (User-agent: OAI-SearchBot, User-agent: Claude-SearchBot, User-agent: PerplexityBot) with the paths you want read.
2. Put training crawlers in a separate group if you want to opt out of training. Blocking GPTBot does not remove you from ChatGPT search.
3. Check every host you own, including docs, blog and help subdomains, which have their own robots.txt files.
4. Check your CDN and firewall for AI bot or bot management settings. They can refuse a crawler that robots.txt allows.
5. Re-run the checker on the new file before you publish it.

Frequently asked questions

How do I check if my website blocks GPTBot? Open your-site/robots.txt, paste its contents into the checker above and read the OpenAI card. GPTBot is OpenAI's training crawler; the search row shows OAI-SearchBot, which decides whether ChatGPT search can show your pages. The two are separate settings, so check both.

Does blocking GPTBot remove my site from ChatGPT? No. OpenAI documents GPTBot as the training control and OAI-SearchBot as the crawler for ChatGPT search answers. You can block GPTBot and still allow OAI-SearchBot, which is what 15 of 569 startup sites in our report do (2.6%; 95% interval 1.6% to 4.3%).

What is ClaudeBot, and should I block it? ClaudeBot is Anthropic's training crawler. Claude-SearchBot is the one that indexes pages for search in Claude, and Anthropic says disabling it may reduce a site's visibility in user search results. Block ClaudeBot only if you want to opt out of training, and keep Claude-SearchBot allowed.

If the checker says "allowed", will AI engines cite my site? Not by itself. Being allowed only means the crawler is not asked to stay away. Whether an engine reads, trusts and cites your page depends on what the page says and how it compares with other sources. Lenz, our free AI visibility analysis, shows how ChatGPT and Gemini answer your buyers' questions today.

Does the checker look at llms.txt or my firewall? No. It reads robots.txt, the file crawlers check first. Firewalls, CDN bot settings and rate limits act at the network edge and cannot be read from a browser. Our llms.txt guide explains what that other file does and includes a generator.

Work with us

There are two ways to work with us. Teams that want the work done for them can use our managed AEO services: Book a call. Teams that want to see where they stand first can start with Lenz, our free analysis of how ChatGPT and Gemini answer their buyers' questions: Get your free AI visibility report.

How this tool was made

Elementera built this checker for the AI Crawler Access Report, which used the same parsing rules on 600 websites. Mojtaba Ebadi wrote this page, Soroush Mehri reviewed it, and Mojtaba Ebadi approved it for publication. Crawler names come from each vendor's documentation, linked below.

Sources

• OpenAI, "Overview of OpenAI Crawlers": developers.openai.com/api/docs/bots
• Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?": support.claude.com
• Perplexity, "Perplexity Crawlers": docs.perplexity.ai
• Google, "Google's common crawlers": developers.google.com
• Bing, "Which crawlers does Bing use?": bing.com/webmasters
• Koster, Illyes, Zeller and Sassman, "Robots Exclusion Protocol", RFC 9309, IETF, 2022: rfc-editor.org