PerfBeeBot, PerfBee's crawler, checks your pages at one of two speeds. This page explains both, how PerfBeeBot treats your robots.txt and what a scan costs.
Two crawl speeds
Polite (every site, by default)
- About two requests per second to a host, with at most two at a time.
- Sites on hosted platforms such as Shopify, Wix and Squarespace get one request per second. All Shopify stores share that pace, because Shopify judges crawlers across its whole platform.
- When your server answers 429 or 503, PerfBeeBot waits as long as
Retry-Afterasks, then slows down. If your site asks for a wait longer than five minutes, the scan stops. - If most requests are refused, the scan pauses (for about 10, then 20, then 40 minutes) and continues more slowly after each pause. If the site still refuses after the last pause, the scan stops. It never switches to another identity to get past a block.
Fast lane (sites that let PerfBeeBot in)
When your firewall allows PerfBeeBot, we can switch your site to the fast lane on request. It sends about five requests per second to your site and does not apply robots.txt Crawl-delay. It still backs off when your server answers 429 or 503, and slows down when your firewall starts refusing requests. See Adding Your Site and Letting PerfBeeBot In for the steps.
On Business and up you can also turn the fast lane on yourself: verify the site, then pass a 10-second speed test. That fast lane keeps following your Crawl-delay. See Fast Lane: Faster Scans of Your Own Site.
Robots.txt
PerfBeeBot crawls the host your scan starts on. If the start address redirects, for example from the bare domain to www, it follows that redirect first. On that host it reads robots.txt, uses the group named PerfBeeBot and falls back to the * group. Links to other hosts, including your other subdomains, are only checked, never crawled; on those hosts PerfBeeBot follows only a group that names PerfBeeBot.
- Pages your robots.txt disallows are not fetched. The report lists them as Not checked.
Crawl-delayandRequest-rateare followed between pages on the polite speed, up to 10 seconds. A longer delay counts as 10 seconds. Images, CSS and scripts on the same host are checked at most once per second while a delay is set.- If robots.txt itself answers with a server error, PerfBeeBot treats the whole host as disallowed, as the robots.txt standard (RFC 9309) asks.
Your own site: Ignore robots.txt
On every plan, you can scan your own site without its robots.txt, for example a staging site that disallows everything. Verify the site first (see Verifying Your Site), then tick Ignore robots.txt in the scan form. The option applies to that scan and to that site only: PerfBeeBot still follows Crawl-delay, and other hosts' robots.txt files as usual. The report notes that the scan skipped robots.txt and how many URLs it disallows were crawled.
A long Crawl-delay makes scans slow. At 10 seconds, PerfBeeBot checks about 360 pages an hour. A scan has a time limit; when it runs out, the scan stops and reports the pages it reached. When your robots.txt sets a delay, the scan form tells you before you start how long the scan will take. To let PerfBee crawl your own site faster, add a PerfBeeBot group without a delay, as described in the setup guide.
Not checked pages
A page PerfBeeBot could not see is not a broken link. The report marks it Not checked and names the reason: robots.txt, a firewall such as Cloudflare or Akamai, rate limiting, a timeout or no answer. Not checked pages cost no credits.
Verified sites
A site you verified in My Sites (see Verifying Your Site), or one PerfBee allowlisted, counts as your own site. Its scans cost the same credits as any other site's. Only add sites you own or are allowed to audit.
On the Free plan, one crawl of a site you have not verified reads at most 500 pages. A site you verified has no such limit on any plan.
| Scan | Credits |
|---|---|
| Fast scan, per page | 1 credit |
| Detailed scan, per page | 1 credit |
| JavaScript rendering, per rendered page | 2 credits |
With smart JavaScript rendering, only the pages that need it are rendered. The other pages cost the scan's normal price. Pages PerfBeeBot could not check cost nothing.