Key point. Google-Extended is Google's robots.txt control token for AI training. It controls whether Google may use content it crawls to train Gemini models and to ground Gemini answers, per Google's page quoted below. It is itself a robots.txt rule and never visits your site.
Google-Extended at a glance
These are the facts our registry holds for Google-Extended, checked against Google's documentation on October 14, 2026.
| Vendor | |
| Purpose | AI training |
| Type | Control token |
| Follows robots.txt | Not applicable: a control token never visits |
| Vendor documentation | Google crawler page, checked October 14, 2026 |
| Published IP ranges | None listed in our registry |
What is Google-Extended?
Google-Extended is Google's robots.txt control token for AI training. It controls whether Google may use content it crawls to train Gemini models and to ground Gemini answers, per Google's page quoted below. Our registry notes: A robots.txt control token, not a separate crawler. Google says it does not affect Google Search.
In Google's own words (Google's crawler documentation, read October 14, 2026):
- “Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.”
- “Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”
- “Google-Extended doesn't have a separate HTTP request user agent string.”
Google uses other names for other jobs: Googlebot (web search); Google-CloudVertexBot (other); GoogleOther (other). Each takes its own robots.txt group.
Does Google-Extended follow robots.txt?
Google-Extended is itself a robots.txt rule rather than a visitor: it has no crawler of its own, and Google reads the group that names it to decide how it may use pages its crawler fetched.
A robots.txt rule records your request. It does not prove a crawler honours it, and firewall or CDN bot settings can refuse a crawler that robots.txt allows, so check those too.
How do I block Google-Extended in robots.txt?
Add a group that names Google-Extended and disallows the whole site:
User-agent: Google-Extended
Disallow: /To keep it out of one folder only, disallow that path instead, for example Disallow: /private/.
Because Google-Extended is a control token, this rule does not stop any crawl; it tells Google how it may use the pages its crawler fetches.
How do I allow Google-Extended?
Google-Extended is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. A crawler obeys the most specific group that names it and falls back to User-agent: * only when no group names it (RFC 9309), so if a catch-all rule blocks crawlers, add a group for it:
User-agent: Google-Extended
Allow: /How can I tell if a request really comes from Google-Extended?
You cannot see it in your logs, because it never visits. Google says: “Google-Extended doesn't have a separate HTTP request user agent string.” The crawling itself is done with Google's existing user agents.
How often do startup websites block Google-Extended?
Our AI Crawler Access Report read the robots.txt files of 600 sampled startup websites on October 7, 2026. This is its row for Google-Extended, across the 569 sites whose answer it could read:
| Crawler | Type | Blocked on at least one path | Blocked on all six paths | Named in the file |
|---|---|---|---|---|
| Google-Extended | AI training | 5 of 569 sites (0.9%; 95% interval 0.4% to 2.0%) | 3 of 569 sites (0.5%; 95% interval 0.2% to 1.5%) | 85 of 569 sites (15%; 95% interval 12% to 18%) |
“Blocked” means the file disallows the crawler on at least one of six common paths (/, /pricing, /blog/, /docs/, /about, /product).
The report's headline figure, for the three AI search crawlers it tracked: “3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) block at least one AI search crawler (Elementera AI Crawler Access Report, October 2026).”
The figures describe that sample of Y Combinator-backed software companies on the day we fetched it. The method, the intervals and the full table are in the AI Crawler Access Report.
What the AI Crawler Access Report adds about Google-Extended:
Outside research points the same way for Google. A 2026 study of Google Search, Gemini and AI Overviews found that sites blocking Google's AI crawler were less likely to be retrieved by AI Overviews (Grossman et al., ACM SIGIR 2026, arXiv 2604.27790). The authors measured a correlation, not a cause, and Google states that Google-Extended does not affect inclusion in Google Search.
How do I check whether my robots.txt blocks Google-Extended?
Paste your robots.txt into our free checker and it shows, crawler by crawler, whether Google-Extended may read the path you choose. Nothing you paste leaves the page.