Skip to content
Crawler

PerfBeeBot

The crawler behind PerfBee: how it identifies itself, how to verify it, how fast it crawls, and how to slow it down, block it or let it in.

Last updated: October 12, 2026

User agent
Mozilla/5.0 (compatible; PerfBeeBot/1.0; +https://perfbee.com/bot)
IP address
169.58.21.218
Reverse DNS
crawler.perfbee.com
Signature-Agent
"https://perfbee.com"
robots.txt token
PerfBeeBot
In short

PerfBeeBot is the crawler behind the PerfBee Site Crawler. It visits a site because someone asked PerfBee to scan it, or for PerfBee's own research (see Research crawls below). It checks links, SEO tags and page content. It does not collect content to train AI models. It does not try to get past a firewall that refuses it.

How to recognise it

  • User agent: Mozilla/5.0 (compatible; PerfBeeBot/1.0; +https://perfbee.com/bot)
  • Pages that need JavaScript are loaded in a headless Chromium. Its user agent ends in (compatible; PerfBeeBot/1.0; +https://perfbee.com/bot).
  • Video and audio links are checked with a single HEAD request, with the same user agent.
  • Requests carry a From: [email protected] header.

How to verify it

  • Crawls, link checks and the free tools on this site sign their requests with Web Bot Auth (HTTP Message Signatures). Each request carries Signature-Agent: "https://perfbee.com" and a signature made with the key published at https://perfbee.com/.well-known/http-message-signatures-directory. A firewall or CDN that supports Web Bot Auth can check it.
  • Requests come from 169.58.21.218. The list is also published as JSON at perfbee.com/bot/ips.json.
  • The address resolves back to crawler.perfbee.com, and crawler.perfbee.com resolves to it.

How fast it crawls

  • At most 2 requests per second to one host. The limit is shared by every scan, so two customers scanning the same site do not double it.
  • Pages that need JavaScript load in Chromium the way a visitor's browser loads them, except that each script and style file is fetched once per scan and then reused, and pages a visitor might click next (prefetch) are not fetched. The page's other requests, such as API calls, count toward the same limit. Requests to known analytics and tracking services are blocked.
  • Sites on hosted platforms such as Shopify, Wix and Squarespace get at most 1 request per second.
  • It follows Crawl-delay and Request-rate in your robots.txt, up to 10 seconds between pages. A longer delay counts as 10 seconds. If that is still too fast for your server, block PerfBeeBot below or email us.
  • When a server answers 429 or 503 it waits as long as Retry-After asks, or longer each time. If most pages keep being refused, it stops.
  • It goes faster only on sites whose owner has allowlisted it with us.

How to slow it down or block it

Add a group for PerfBeeBot to your robots.txt. This one asks for five seconds between pages and keeps it out of one folder:

robots.txt · slow it down
User-agent: PerfBeeBot
Crawl-delay: 5
Disallow: /private/

A group for PerfBeeBot replaces your User-agent: * group for it. Copy any rule you want it to keep into the new group.

To keep it out of the whole site:

robots.txt · block it
User-agent: PerfBeeBot
Disallow: /

When a page we scan links to your site, PerfBeeBot may send one request to check that the link works. It does not crawl your site from there. These checks follow a robots.txt group for PerfBeeBot, such as the ones above. Your User-agent: * rules apply when your own site is scanned. To have your domain excluded from all scans, email [email protected].

Research crawls

PerfBee also runs its own research, such as a study of what crawlers that do not run JavaScript can read on online shops. A research crawl works like this:

  • It visits a sample of public sites, at most twice a year.
  • It sends at most 8 requests per site for pages, robots.txt and sitemaps. Redirects count too. If a page needs JavaScript, Chromium also loads that page's files within the limits above.
  • It follows your robots.txt. It honors Crawl-delay in full, up to 60 seconds between pages. A site that asks for more is skipped. On a site with a Crawl-delay, files are fetched at most once a second.
  • It stops at the first refusal, such as a 401, a 403 or a challenge page.
  • It uses the same IP address, user agent and signature as every other PerfBeeBot request.
  • It publishes only totals. It never names a site or scores one.

Research crawls differ from customer scans. Our terms limit customer scans to sites the customer owns or may analyze. To keep your site out of research crawls only, email [email protected]. The robots.txt group above keeps PerfBeeBot out of everything.

How to let it in

If you scan your own site with PerfBee and your firewall refuses the crawler, allowlist the IP address above. You can also allow the X-PerfBee-Token header shown for your site in the dashboard. The scan report names the protection that refused a page and shows the steps for it.

Contact

Questions or reports about PerfBeeBot: [email protected].