Elementera
Products
OriginumLenzThe Supervised plan, our managed AEO service
Case StudiesBlogFree ToolsResearchAbout UsBook a call
Book a call
  1. Home
  2. AI crawlers
  3. Bytespider

Bytespider: what it is, and how to allow or block it in robots.txt

Key point. Bytespider is ByteDance's AI training crawler. It governs use of your content for model training. It is reported not to follow robots.txt.

Bytespider at a glance

These are the facts our registry holds for Bytespider. No vendor page documents it.

VendorByteDance
PurposeAI training
TypeUser agent
Follows robots.txtReported not to; source: community ai.robots.txt list (commit 9ad8a47e)
Vendor documentationNone found: no vendor page documents this token
Published IP rangesNone listed in our registry
Seen on elementera.com, October 22 to 31, 2026157 verified requests (2.9% of verified AI crawler requests)

What is Bytespider?

Bytespider is ByteDance's AI training crawler. It governs use of your content for model training. Our registry notes: ai.robots.txt lists it as LLM training and records that it does not respect robots.txt.

What does Bytespider fetch on our own site?

Bytespider fetched 37 different pages of our own site from October 22 to 31, 2026, in 157 verified requests. We log every request through Cloudflare (10 days, sampled counts, so each figure is an estimate):

Bytespider on elementera.comOctober 22 to 31, 2026
Requests naming Bytespider295
Verified as ByteDance's own157: 157 by Cloudflare's verified-bot flag, 0 by ByteDance's published IP list
Name only, not verified138
Bytespider's share of verified AI crawler requests2.9% of 5,447
Verified Bytespider requests per day17, 10, 35, 9, 24, 0, 20, 22, 0, 20
First verified Bytespider request we loggedOctober 22, 2026, 06:03 UTC
Cloudflare files Bytespider under“Search Engine Crawler”
Bytespider came fromCHINANET BACKBONE (AS4134), 57; CHINA UNICOM China169 Backbone (AS4837), 52
User agent Bytespider sent most (157 of 157)Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
Bytespider read our robots.txt0 times
What our site answered Bytespider4xx 157

Bytespider's daily count fell: 95 verified requests in the first five days, 62 in the last five.

Out of 37 different pages of ours, Bytespider fetched these most: / (22), /blog/how-to-do-prompt-research-aeo-ai-search-optimization (19), /blog (14), /blog/will-ai-aeo-replace-search-engines-in-the-next-decade (8), /blog/aeo-vs-seo-measured (6).

By section, Bytespider's page requests went to blog posts 45%, other pages 15%, home page 14%, AI crawler pages 8.5%, case studies 5.2%.

Our own robots.txt names Bytespider in its AI group, which closes 3 old blog posts; none of Bytespider's 157 verified requests asked for one.

Every verified Bytespider request was refused with 403 Forbidden, yet neither our robots.txt nor our Cloudflare zone blocks it. The refusal comes from Webflow, which hosts this site: on October 26, 2026 Webflow's edge answered the Bytespider user agent with 403 on our site, on our Webflow staging host and on webflow.com itself, while GPTBot and an ordinary browser got 200.

The 138 unverified requests that used the name came mostly from Google Cloud (AS396982); 71 of them asked for files that are not pages here, such as /.env.development.local.

Does Bytespider follow robots.txt?

Not reliably: our registry records that Bytespider does not follow robots.txt, from the community ai.robots.txt list (commit 9ad8a47e).

A robots.txt rule records your request. It does not prove a crawler honours it, and firewall or CDN bot settings can refuse a crawler that robots.txt allows, so check those too.

How do I block Bytespider in robots.txt?

Add a group that names Bytespider and disallows the whole site:

User-agent: Bytespider
Disallow: /

To keep it out of one folder only, disallow that path instead, for example Disallow: /private/.

robots.txt may not stop it. It is reported not to follow robots.txt. If you must keep it out, the block has to happen in your firewall or CDN, not only in robots.txt.

How do I allow Bytespider?

Bytespider is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. A crawler obeys the most specific group that names it and falls back to User-agent: * only when no group names it (RFC 9309), so if a catch-all rule blocks crawlers, add a group for it:

User-agent: Bytespider
Allow: /

How can I tell if a request really comes from Bytespider?

Treat the name alone as a claim. User agents can be faked, and our registry lists no published IP-range file for Bytespider, so there is no list to match the request's address against.

How do I block requests that only pretend to be Bytespider?

To block requests that only pretend to be Bytespider, use Cloudflare's verified-bot flag: on our site it flagged 157 of 157 verified requests. Add a custom rule with this expression and the action Block:

(http.user_agent contains "Bytespider" and not cf.client.bot)

On our October 22 to 31, 2026 logs it would have stopped the 138 unverified requests and let all 157 verified ones through.

On a site hosted by Webflow you need no rule: Webflow already refuses Bytespider, real or not.

How often do startup websites block Bytespider?

Our AI Crawler Access Report read the robots.txt files of 600 sampled startup websites on October 7, 2026. This is its row for Bytespider, across the 569 sites whose answer it could read:

CrawlerTypeBlocked on at least one pathBlocked on all six pathsNamed in the file
BytespiderAI training15 of 569 sites (2.6%; 95% interval 1.6% to 4.3%)12 of 569 sites (2.1%; 95% interval 1.2% to 3.6%)38 of 569 sites (6.7%; 95% interval 4.9% to 9.0%)

“Blocked” means the file disallows the crawler on at least one of six common paths (/, /pricing, /blog/, /docs/, /about, /product).

The report's headline figure, for the three AI search crawlers it tracked: “3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) block at least one AI search crawler (Elementera AI Crawler Access Report, October 2026).”

The figures describe that sample of Y Combinator-backed software companies on the day we fetched it. The method, the intervals and the full table are in the AI Crawler Access Report.

What the AI Crawler Access Report adds about Bytespider:

Bytespider has the highest share of blocks among the crawlers we tracked: 15 of 569 sites (2.6%; 95% interval 1.6% to 4.3%). Its interval overlaps those of 9 other crawlers, so the order among them is not settled at this sample size.

How do I check whether my robots.txt blocks Bytespider?

Read the group by hand: our free crawler checker covers 16 crawlers, and Bytespider is not one of them yet.

  1. Open yoursite.com/robots.txt.
  2. Look for a User-agent: Bytespider line. If there is one, only that group applies to Bytespider.
  3. If there is none, the User-agent: * group applies.
  4. Within the group, the longest matching Allow or Disallow rule for a path wins.

Where can I see every AI crawler?

Our list of AI crawlers shows all 29 names in our registry, filtered by purpose. The same data is a free JSON download in our AI crawler registry.

Frequently asked questions

Three questions site owners ask about this name, each answered in one paragraph.

Should I block Bytespider?

That is a choice about model training, not about search. Blocking Bytespider does not remove your pages from AI search engines, whose crawlers have their own names.

Does blocking Bytespider affect other crawlers?

No. robots.txt groups are matched by name, so a group for Bytespider applies only to Bytespider.

How do I see whether Bytespider visits my site?

Search your server or CDN logs for Bytespider in the user agent. User agents can be faked, and our registry lists no published IP-range file for it, so treat the name alone as a claim.

Work with us

There are two ways to work with us. Teams that want the work done for them can use our managed AEO services: Book a call. Teams that want to see where they stand first can start with Lenz, our free analysis of how ChatGPT and Gemini answer their buyers' questions: Get your free AI visibility report.

How this page was made

Written by Mojtaba Ebadi. Reviewed by Soroush Mehri.

Sources

  • Elementera, Cloudflare request logs for elementera.com, October 22 to 31, 2026 (sampled counts)
  • ai.robots.txt, community list of AI agents: github.com/ai-robots-txt/ai.robots.txt
  • Koster, Illyes, Zeller and Sassman, “Robots Exclusion Protocol”, RFC 9309, IETF, 2022: rfc-editor.org
  • Elementera, AI Crawler Access Report, October 2026: /resources/ai-crawler-access-report
Elementera
Expertise in. Original content out.

Elementera turns what a business knows into original content for AI search, and measures how AI engines describe and recommend it.

Product
OriginumManaged AEO ServicesLenzBook a Call
Learn
BlogCase StudiesFree ToolsResourcesResearchEvents
Guides
What is AEOAEO vs SEOAI visibility funnelllms.txt guide
Company
About UsContactSite map
hello@elementera.comTrustPrivacy PolicyTerms of Service

Supported by

Google
Microsoft
NVIDIA
Cloudflare
Anthropic

Machine-readable summary: llms.txt · Company facts