Key point. Googlebot is Google's search engine crawler for Google Search, including its AI features. Blocking it removes your pages from that search engine, including AI answers built on its index. It follows robots.txt.
Googlebot at a glance
These are the facts our registry holds for Googlebot, checked against Google's documentation on October 14, 2026.
| Vendor | |
| Used by | Google Search, including its AI features |
| Purpose | Web search |
| Type | User agent |
| Follows robots.txt | Yes; source: vendor's documentation, as recorded in our AI crawler check |
| Vendor documentation | Google crawler page, checked October 14, 2026 |
| Published IP ranges | developers.google.com/static/crawling/ipranges/common-crawlers.json |
What is Googlebot?
Googlebot is Google's search engine crawler for Google Search, including its AI features. Blocking it removes your pages from that search engine, including AI answers built on its index. Our registry notes: Also the crawler for Google AI Overviews and AI Mode, which draw on the Google Search index.
In Google's own words (Google's crawler documentation, read October 14, 2026):
- “Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Video mode, Google News, and Discover.”
Google uses other names for other jobs: Google-Extended (AI training); Google-CloudVertexBot (other); GoogleOther (other). Each takes its own robots.txt group.
Does Googlebot follow robots.txt?
Yes, Googlebot follows robots.txt: our registry records this from the vendor's documentation, as recorded in our AI crawler check.
What Google says about its common crawlers (Google's crawler documentation, read October 14, 2026):
- “They always obey robots.txt rules when crawling automatically.”
A robots.txt rule records your request. It does not prove a crawler honours it, and firewall or CDN bot settings can refuse a crawler that robots.txt allows, so check those too.
How do I block Googlebot in robots.txt?
Add a group that names Googlebot and disallows the whole site:
User-agent: Googlebot
Disallow: /To keep it out of one folder only, disallow that path instead, for example Disallow: /private/.
How do I allow Googlebot?
Googlebot is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. A crawler obeys the most specific group that names it and falls back to User-agent: * only when no group names it (RFC 9309), so if a catch-all rule blocks crawlers, add a group for it:
User-agent: Googlebot
Allow: /Want the exact block for every AI crawler at once? Our robots.txt generator for AI crawlers writes it for each choice.
How can I tell if a request really comes from Googlebot?
Match the request's IP address against Google's published list, because user agents can be faked: https://developers.google.com/static/crawling/ipranges/common-crawlers.json.
How often do startup websites block Googlebot?
Our AI Crawler Access Report read the robots.txt files of 600 sampled startup websites on October 7, 2026. This is its row for Googlebot, across the 569 sites whose answer it could read:
| Crawler | Type | Blocked on at least one path | Blocked on all six paths | Named in the file |
|---|---|---|---|---|
| Googlebot | Web search (reference) | 3 of 569 sites (0.5%; 95% interval 0.2% to 1.5%) | 0 of 569 sites (0%; 95% interval 0% to 0.7%) | 35 of 569 sites (6.2%; 95% interval 4.5% to 8.4%) |
“Blocked” means the file disallows the crawler on at least one of six common paths (/, /pricing, /blog/, /docs/, /about, /product).
The report's headline figure, for the three AI search crawlers it tracked: “3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) block at least one AI search crawler (Elementera AI Crawler Access Report, October 2026).”
The figures describe that sample of Y Combinator-backed software companies on the day we fetched it. The method, the intervals and the full table are in the AI Crawler Access Report.
What the AI Crawler Access Report adds about Googlebot:
0 of 569 sites (0%; 95% interval 0% to 0.7%) close the whole site to crawlers, Googlebot and Bingbot included.
Microsoft Copilot and Google's AI features draw on the Bing and Google indexes, so for them Bingbot and Googlebot are the crawlers that matter.
How do I check whether my robots.txt blocks Googlebot?
Paste your robots.txt into our free checker and it shows, crawler by crawler, whether Googlebot may read the path you choose. Nothing you paste leaves the page.