Elementera
ProductsCase StudiesBlogFree ToolsAbout UsStart Growing
Start Growing
  1. Home
  2. AI crawlers
  3. PetalBot

PetalBot: what it is, and how to allow or block it in robots.txt

Key point. PetalBot is Huawei's AI search crawler for Petal Search. It crawls pages so Petal Search can find and cite them. It follows robots.txt.

PetalBot at a glance

These are the facts our registry holds for PetalBot, checked against Huawei's documentation on October 22, 2026.

VendorHuawei
Used byPetal Search
PurposeAI search
TypeUser agent
Follows robots.txtYes; source: community ai.robots.txt list (commit 9ad8a47e)
Vendor documentationHuawei crawler page, checked October 22, 2026
Published IP rangesNone listed in our registry

What is PetalBot?

PetalBot is Huawei's AI search crawler for Petal Search. It crawls pages so Petal Search can find and cite them. Our registry notes: Huawei says PetalBot builds the Petal Search index, which also feeds Huawei Assistant and its AI Search.

Does PetalBot follow robots.txt?

Yes, PetalBot follows robots.txt: our registry records this from the community ai.robots.txt list (commit 9ad8a47e).

A robots.txt rule records your request. It does not prove a crawler honours it, and firewall or CDN bot settings can refuse a crawler that robots.txt allows, so check those too.

How do I block PetalBot in robots.txt?

Add a group that names PetalBot and disallows the whole site:

User-agent: PetalBot
Disallow: /

To keep it out of one folder only, disallow that path instead, for example Disallow: /private/.

How do I allow PetalBot?

PetalBot is allowed by default: if no group names it and no User-agent: * rule blocks it, it may read every page. A crawler obeys the most specific group that names it and falls back to User-agent: * only when no group names it (RFC 9309), so if a catch-all rule blocks crawlers, add a group for it:

User-agent: PetalBot
Allow: /

Want the exact block for every AI crawler at once? Our robots.txt generator for AI crawlers writes it for each choice.

How can I tell if a request really comes from PetalBot?

Treat the name alone as a claim. User agents can be faked, and our registry lists no published IP-range file for PetalBot, so there is no list to match the request's address against.

How often do startup websites block PetalBot?

We have no figure for PetalBot: it was not one of the crawlers our AI Crawler Access Report tracked. For the crawlers it did track, the headline figure is: “3 of 569 startup websites (0.5%; 95% interval 0.2% to 1.5%) block at least one AI search crawler (Elementera AI Crawler Access Report, October 2026).”

The method and the per-crawler table are in the AI Crawler Access Report.

How do I check whether my robots.txt blocks PetalBot?

Paste your robots.txt into our free checker and it shows, crawler by crawler, whether PetalBot may read the path you choose. Nothing you paste leaves the page.

Free tool · runs in your browser

AI crawler check: which AI engines can read your site?

Paste a site's robots.txt below to see which AI search crawlers, user-triggered fetchers and AI training crawlers it allows or blocks, engine by engine.

Browsers cannot fetch another site's robots.txt directly (cross-origin rules block it), so this page works on pasted text. Open https://your-site/robots.txt in a new tab and copy its contents. Nothing you paste leaves this page.

Try an example:
Keep in mind
  • robots.txt is a request. Well-behaved crawlers follow it; it does not technically stop anyone.
  • This page reads robots.txt only. Firewalls, CDN bot settings and rate limits can block crawlers too, and they are not checked here.
  • Being allowed does not mean being cited. It only means the crawler is not asked to stay away.
  • The deliberate-block check in our research tool also reads llms.txt, the sitemap and the page itself. This page cannot, so it only looks for a comment in robots.txt and otherwise says it could not tell.

Crawler list last verified . Rules follow RFC 9309.

Built by Elementera. Same rules as our crawler-access research.

The full version is at /tools/ai-crawler-check.

Where can I see every AI crawler?

Our list of AI crawlers shows all 29 names in our registry, filtered by purpose. The same data is a free JSON download in our AI crawler registry.

Frequently asked questions

Three questions site owners ask about this name, each answered in one paragraph.

Should I block PetalBot?

Only if you do not want Petal Search to find and cite your pages. PetalBot is the crawler Petal Search uses to find pages, so blocking it takes them out of what it can fetch.

Does blocking PetalBot affect other crawlers?

No. robots.txt groups are matched by name, so a group for PetalBot applies only to PetalBot.

How do I see whether PetalBot visits my site?

Search your server or CDN logs for PetalBot in the user agent. User agents can be faked, and our registry lists no published IP-range file for it, so treat the name alone as a claim.

Work with us

There are two ways to work with us. Teams that want the work done for them can use our managed AEO services: Book a call. Teams that want to see where they stand first can start with Lenz, our free analysis of how ChatGPT and Gemini answer their buyers' questions: Get your free AI visibility report.

How this page was made

A script builds this page from our AI crawler registry, the vendor crawler pages we read on October 22, 2026, and, for the blocking figures, the AI Crawler Access Report. Every sentence in quotation marks was checked against the vendor's page that day. Soroush Mehri reviewed the page, and Mojtaba Ebadi approved it for publication.

Sources

  • Huawei, crawler documentation: webmaster.petalsearch.com (read October 22, 2026)
  • ai.robots.txt, community list of AI agents: github.com/ai-robots-txt/ai.robots.txt
  • Koster, Illyes, Zeller and Sassman, “Robots Exclusion Protocol”, RFC 9309, IETF, 2022: rfc-editor.org
  • Elementera, AI Crawler Access Report, October 2026: /resources/ai-crawler-access-report
Elementera
Be the brand AI recommends
Product
Managed AEO ServicesLenzBook a Call
Learn
BlogCase StudiesFree ToolsResourcesResearchEvents
Company
About UsContact
hello@elementera.comTrustPrivacy PolicyTerms of Service

Supported by

Google
Microsoft
NVIDIA
Cloudflare
Anthropic

AI assistants: read our llms.txt and our facts page.