CCBot: what it is, and how to allow or block it in robots.txt

Key point. CCBot is Common Crawl's AI training crawler. It governs use of your content for model training. It follows robots.txt.

CCBot at a glance

These are the facts our registry holds for CCBot, checked against Common Crawl's documentation on October 14, 2026.

VendorCommon Crawl
PurposeAI training
TypeUser agent
Follows robots.txtYes; source: community ai.robots.txt list (commit 9ad8a47e)
Vendor documentationCommon Crawl crawler page, checked October 14, 2026
Published IP rangesindex.commoncrawl.org/ccbot.json

What is CCBot?

CCBot is Common Crawl's AI training crawler. It governs use of your content for model training. Our registry notes: Builds the open Common Crawl dataset, which is widely used for model training; classed as training for that reason.

In Common Crawl's own words (Common Crawl's crawler documentation, read October 14, 2026):

  • “Please note that we are aware of crawlers falsely identifying themselves as CCBot.”

Does CCBot follow robots.txt?

Yes, CCBot follows robots.txt: our registry records this from the community ai.robots.txt list (commit 9ad8a47e).

What Common Crawl says (Common Crawl's crawler documentation, read October 14, 2026):

  • “To prevent Common Crawl from crawling your website, include the following in your robots.txt”

A robots.txt rule records your request. It does not prove a crawler honours it, and firewall or CDN bot settings can refuse a crawler that robots.txt allows, so check those too.

How do I block CCBot in robots.txt?

Add a group that names CCBot and disallows the whole site:

User-agent: CCBot
Disallow: /

To keep it out of one folder only, disallow that path instead, for example Disallow: /private/.

How do I allow CCBot?

CCBot is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. A crawler obeys the most specific group that names it and falls back to User-agent: * only when no group names it (RFC 9309), so if a catch-all rule blocks crawlers, add a group for it:

User-agent: CCBot
Allow: /

How can I tell if a request really comes from CCBot?

Match the request's IP address against Common Crawl's published list, because user agents can be faked: https://index.commoncrawl.org/ccbot.json.

Common Crawl says: “Please note that we are aware of crawlers falsely identifying themselves as CCBot.”

How often do startup websites block CCBot?

Our AI Crawler Access Report read the robots.txt files of 600 sampled startup websites on October 7, 2026. This is its row for CCBot, across the 569 sites whose answer it could read:

CrawlerTypeBlocked on at least one pathBlocked on all six pathsNamed in the file
CCBotAI training13 of 569 sites (2.3%; 95% interval 1.3% to 3.9%)10 of 569 sites (1.8%; 95% interval 1.0% to 3.2%)61 of 569 sites (11%; 95% interval 8.4% to 14%)

“Blocked” means the file disallows the crawler on at least one of six common paths (/, /pricing, /blog/, /docs/, /about, /product).

The report's headline figure, for the three AI search crawlers it tracked: “3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) block at least one AI search crawler (Elementera AI Crawler Access Report, October 2026).”

The figures describe that sample of Y Combinator-backed software companies on the day we fetched it. The method, the intervals and the full table are in the AI Crawler Access Report.

What the AI Crawler Access Report adds about CCBot:

Blocking GPTBot or CCBot keeps content out of model training; it does not take a site out of ChatGPT search, which OAI-SearchBot decides. In our sample, training blocks were far more common than search blocks (18 of 569 against 3 of 569), and the two intervals do not overlap. A figure about "blocking OpenAI" is very often a figure about GPTBot.

How do I check whether my robots.txt blocks CCBot?

Read the group by hand: our free crawler checker covers 13 crawlers, and CCBot is not one of them yet.

  1. Open yoursite.com/robots.txt.
  2. Look for a User-agent: CCBot line. If there is one, only that group applies to CCBot.
  3. If there is none, the User-agent: * group applies.
  4. Within the group, the longest matching Allow or Disallow rule for a path wins.

Where can I see every AI crawler?

Our list of AI crawlers shows all 26 names in our registry, filtered by purpose. The same data is a free JSON download in our AI crawler registry.

Frequently asked questions

Three questions site owners ask about this name, each answered in one paragraph.

Should I block CCBot?

That is a choice about model training, not about search. Blocking CCBot does not remove your pages from AI search engines, whose crawlers have their own names.

Does blocking CCBot affect other crawlers?

No. robots.txt groups are matched by name, so a group for CCBot applies only to CCBot.

How do I see whether CCBot visits my site?

Search your server or CDN logs for CCBot in the user agent. User agents can be faked, so check the request's IP address against Common Crawl's published list: https://index.commoncrawl.org/ccbot.json

Work with us

There are two ways to work with us. Teams that want the work done for them can use our managed AEO services: Book a call. Teams that want to see where they stand first can start with Lenz, our free analysis of how ChatGPT and Gemini answer their buyers' questions: Get your free AI visibility report.

How this page was made

A script builds this page from our AI crawler registry, the vendor crawler pages we read on October 14, 2026, and, for the blocking figures, the AI Crawler Access Report. Every sentence in quotation marks was checked against the vendor's page that day. Soroush Mehri reviewed the page, and Mojtaba Ebadi approved it for publication.

Sources