Crawling a Private Staging Site Through a Tunnel

PerfBee cannot enter your network. PerfBeeBot crawls from the internet, so a staging site that lives only on a private network or behind a VPN is out of its reach. If you can put that site on a public address and let in only requests that carry a secret, PerfBeeBot can crawl it. A tunnel that you run does the first part, and a service token does the second.

This article uses Cloudflare Tunnel and a Cloudflare Access service token as the example, because that token is a pair of request headers. Any tunnel works if it gives the site a public HTTPS address and can require a secret header.

Before you start

• Your plan is Starter or higher. Staging access is part of it.
• You can create a tunnel and add a DNS record for a hostname you control, for example staging.your-site.com. With Cloudflare, the domain must be added to Cloudflare first.
• Your security team agrees to put the site on a public hostname. If it will not, a crawler that runs on your own computer, inside your network, is the right tool. PerfBee does not run inside customer networks.

1. Put the site on a public address

1. Run cloudflared on a machine that can reach your staging app. It opens outbound connections only, so you do not open an inbound port.
2. In Cloudflare, add a public hostname for the tunnel, such as staging.your-site.com, that points to the app's internal address.

The hostname is now public. Do the next step before anything else.

2. Let in only requests with a service token

1. In Cloudflare Access, add the hostname as a self-hosted application.
2. Create a service token. Cloudflare gives you a Client ID and a Client Secret. It shows the secret once, so copy it now.
3. Add an Access policy with the Service Auth action that includes this token.

A public hostname without an Access policy is open to everyone. Check that a request without the token is refused before you go on. Choose an expiry for the token and note the date. Cloudflare's own documentation has the current menus for Tunnel and for service tokens.

3. Add and verify the site in PerfBee

1. Add https://staging.your-site.com to My Sites.
2. Click the site, find the Ownership card and use the DNS record tab. Publish the record, then click Check now. See Verifying Your Site.

Use DNS, not the file or the meta tag. Those two checks fetch a page from the site, and the Access policy stops them. A tunnel hostname is a CNAME, and a name with a CNAME cannot hold a TXT record too, so put the record at _perfbee.staging.your-site.com, the name the card shows. PerfBee checks the record again every week, so leave it in place.

4. Give PerfBeeBot the token

1. Open My Sites, click the site and find the Staging access card.
2. Add two request headers. The first is named CF-Access-Client-Id, with the Client ID as its value. The second is named CF-Access-Client-Secret, with the Client Secret as its value.
3. Save, then run Test access on the same page.

Test access alone does not prove that PerfBeeBot got in. The request behind its Homepage line follows redirects, so a redirect to a sign-in page that answers HTTP 200 still shows Pass. Its robots.txt line is read without the headers and can show Check. To be sure, run a small scan: choose Fast, set Max pages to 10 under Advanced settings and check that the report lists your own pages. If it says robots.txt disallows the crawler, see the last item at the end of this article.

Then scan the site as usual. See Crawling a Staging or Password-Protected Site for how PerfBee sends and keeps the values.

What PerfBeeBot does with the token

• In a scan, it sends the two headers on every page, file and link request to that hostname, including pages rendered with JavaScript. It does not keep cookies, so each request carries them. Two things go without them. The optional Videos, Iframes and Media checks send HEAD requests with no custom header. The first request of a scan only finds where your site's address leads.
• It sends them over HTTPS only, and never to another hostname. A link to another host is checked without them, so if that host is also behind Access, the check cannot get in.
• PerfBee stores the values encrypted and never shows them again, in the dashboard, in reports or in exports. The Staging access card lists only the header names.
• Some headers cannot be set here, because PerfBee sends them itself or they belong to the connection: Host, Cookie, Authorization (use the Basic auth fields instead), User-Agent, From, Connection and the other hop-by-hop headers. The Cloudflare token headers are not among them.
• You can set up to five headers, so you can add others your staging setup needs.

When the token ends

If you delete the token or it expires, PerfBeeBot can no longer get in, and a scan cannot read the site. Create a new token, click Replace in the Staging access card, add both headers again with the new values, save, and check as in step 4.

To stop sending the token, click Remove in the card and confirm. A scheduled scan still runs without the headers, and the protected site turns it away. To stop it, pause or delete the schedule in the Scheduled tab of Site Crawler.

What does not work behind the wall

• PageSpeed tests, including the performance test and the accessibility audits in Page Detail. Google loads the page for those tests, and Google has no token. PerfBee's own page load in the performance test sends no headers either. A protected page gives an error or the score of the block page.
• These tools load a page on their own and send no headers: Compare with another page, Accessibility by template, Critical CSS, Recheck in Site Pages, and Test on a page in the custom extraction and custom search editors of the scan form. The AI Readiness tab of a Detailed scan also loads pages, after the scan ends. On a protected page they get an error or the block page.
• The Sitemap estimate card in the scan form reads without the headers and can say No sitemap found. The scan itself reads the sitemap with them.
• Bot protection in front of the site. Allowlist PerfBeeBot as described in Adding Your Site and Letting PerfBeeBot In.
• Staging robots.txt files that disallow everything. See Crawling a Staging or Password-Protected Site for Ignore robots.txt on a verified site.