Scans of a site you own can go faster: up to five requests a second instead of two. A 10,000-page site then takes about 35 minutes instead of an hour and a half. It comes with Business and up and costs no extra credits.
Turn it on
- Open My Sites and click the site.
- In Ownership, prove the site is yours with a DNS record, a file or a meta tag. See Verifying Your Site.
- In Fast lane, tick the box and click Run the test. PerfBeeBot reads about 40 pages from your sitemap at five requests a second, from its crawler address
169.58.21.218. If no page is refused, slow or broken, the fast lane turns on.
Every scan of that site then uses it, scheduled scans too. When you pick the site in the scan form, it says Fast lane.
When the test does not pass
The card says why. Most often:
- A page was refused (401, 403, 429, 503 or a bot wall). Allowlist PerfBeeBot in your firewall or host, then run the test again.
- Your robots.txt asks PerfBeeBot to wait between pages (
Crawl-delayorRequest-rate). Scans keep following it, so remove it to use the fast lane. - Pages took more than two seconds to answer (the median), or more than two pages did not answer.
- The site runs on a hosted platform such as Shopify, which sets one pace for every store on it.
- Your sitemap has fewer than 10 pages that robots.txt lets PerfBeeBot read.
What stays the same
Only the site you verified goes faster. Its subdomains and every other site keep the normal pace. PerfBeeBot still follows robots.txt, slows down when your server answers 429 or 503, and pauses when requests are refused. Pages that need JavaScript still render two at a time.
When it turns itself off
If a scan is refused at this pace, the fast lane turns off and you get a notification and an email. It also turns off when the site's verification is lost. Run the test again to turn it back on. You can turn it off yourself at any time.