Key point. Block AI training without leaving AI search: block the training crawlers (GPTBot, ClaudeBot, Google-Extended and others) and keep the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed. This tool writes that block for you.
robots.txt generator for AI crawlers
Free tool · runs in your browser
robots.txt AI policy builder
Choose which AI crawlers may read your site. Start from a preset, adjust each crawler, then copy the block into your robots.txt. Each crawler is listed with its purpose: AI search, training, or fetching a page when a person asks.
In plain English
robots.txt is a request, not enforcement. Well-behaved crawlers read it and follow it. Nothing stops a crawler that ignores it, and some vendors say their assistants may not apply it when a person asks for a page. To block a crawler for certain, block it at your server or CDN by user agent and verified IP range.
Paste the block near the top of your robots.txt, keep your existing rules below it, then test the full file with our AI crawler check.
Crawler list checked against each vendor's documentation on 2026-10-14. Download the AI crawler registry (JSON).
Which AI crawlers should I block?
Each AI vendor runs separate crawlers for separate jobs. A search crawler reads your pages so the vendor's AI search can find and cite them. A training crawler, or a control token such as Google-Extended, decides whether your content may be used to train models. A user fetcher reads one page when a person asks an assistant to look at it.
So the question is not "AI or no AI". It is three questions per vendor. Most businesses that want to be found in AI answers keep search allowed. Whether to allow training is a policy choice: blocking it does not remove you from AI search results.
What do the three presets do?
- Allow all AI writes no blocks. Every crawler follows your existing
User-agent: *rules. - Allow AI search, block training blocks the training crawlers and control tokens only.
- Block all AI crawlers blocks every AI crawler. Googlebot, Bingbot and Applebot stay allowed, because blocking them removes you from those search engines, and Google's AI Overviews and Microsoft Copilot draw on the same index.
Does robots.txt actually stop AI crawlers?
robots.txt is a request. Well-behaved crawlers follow it; nothing enforces it. OpenAI and Perplexity say their user fetchers may not apply it when a person asks for a page. To block a crawler for certain, block it at your server or CDN by user agent and verified IP range. Our AI crawler registry lists the published IP ranges.
After you paste the block, test the whole file with the AI crawler check.
FAQ
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, which you can keep allowed.
Should I block Google-Extended?
Google-Extended is a control token, not a crawler. Google says it does not affect Google Search. Blocking it asks Google not to use your content for training its Gemini models.
Can I block Google AI Overviews in robots.txt?
Not without blocking Googlebot, which removes you from Google Search. AI Overviews are part of Search and use the same crawler.
Why are my allowed crawlers not listed in the block?
Crawlers without their own group follow your User-agent: * rules. Tick "Write Allow groups" only if your * rules block pages you want AI search to read; a named group replaces the * rules for that crawler.
Work with us
Want it done for you? Book a call about managed AEO services. Want to see where you stand first? Get your free AI visibility report from Lenz.