A staging site is often behind a password or a protection header. PerfBeeBot can crawl it when you give it the values it needs. They are sent only to that site.
Before you start
- The site is in My Sites and verified. See Verifying Your Site.
- Your plan is Starter or higher.
- The site can be reached from the internet. PerfBeeBot crawls from
169.58.21.218. If your staging site only lets in some addresses, add this one.
Add staging access
- Open My Sites, click the site and find the Staging access card.
- Enter a Basic auth user name and Password, or add up to five request headers, with a Header name and a Value each. A Vercel protection bypass header is one example.
- Save, then run Test access on the same page to check that PerfBeeBot gets in.
Where PerfBeeBot uses them
Scans of that site use the values, including scheduled scans and pages rendered with JavaScript. So do the robots.txt and sitemap reads, and Test access. PerfBeeBot sends them only:
- to that site (
www.and the bare domain count as one site), never to other hosts or to links that leave the site; - over HTTPS, never over plain HTTP.
How the values are kept
PerfBee stores the values encrypted. After you save them, the dashboard shows only the user name, partly hidden, and the header names, never a password or a header value. They do not appear in reports or exports. To change them, enter them again. To stop, click Remove staging access. PerfBeeBot stops sending the values to the site.
Robots.txt on staging sites
Many staging sites disallow everything in their robots.txt to stay out of search engines. PerfBeeBot follows that too, so a scan would read nothing. For your own verified site you can tick Ignore robots.txt in the scan form. Crawl-delay still applies.
What it does not do
- It does not sign in through a login form. Only Basic auth and request headers are supported.
- It does not get past bot protection such as Cloudflare's. Allowlist PerfBeeBot as described in Adding Your Site and Letting PerfBeeBot In.
- It cannot reach a site that is only on a private network or behind a VPN.