Skip to content
Site Crawler

Advanced Scan Settings

3 min read · Updated Nov 22, 2026

Fine-tune your PerfBee Site Crawler scans with advanced settings. Control which URLs are crawled and whether to include external links and resources.

Advanced Scan Settings

The PerfBee Site Crawler offers a range of advanced settings to give you granular control over your website scans. These options allow you to tailor the crawl to your specific needs, whether you're focused on internal links, external validation, or which parts of your site get crawled. These settings are configured inside a preset or directly in the scan form on the Site Crawler's Scan tab, most of them under Advanced settings.

Excluding URLs

Sometimes you need to exclude certain sections of your site from the crawl. This is done with the Exclude URLs field. Enter one URL pattern per line, such as /admin/* or /cart/*. The * matches anything; a pattern without it matches only that exact URL. Excluded pages are not crawled or billed, so this helps to avoid crawling unnecessary areas and saves credits.

Including only some URLs

The Include URLs field, next to Exclude URLs, limits the crawl to the pages that match. Enter one pattern per line, such as /en/* or /de/*. The patterns work like Exclude patterns. With patterns set, PerfBee crawls only the pages of your site that match one of them. Your other pages are still checked when a crawled page links to them, so a dead link to /blog/ shows up, but they are not crawled, not analysed and not billed. When a page matches both lists, Exclude wins and the page is not requested at all. The start page is always read. Leave the field empty to crawl every page, and note that a List of URLs scan names its pages itself and ignores Include. The credit estimate counts the pages Include leaves out as "left out".

Page caps

Under Limits, Page caps reads at most a set number of the pages that match a URL pattern, for example 200 pages of /products/*. Click Add cap, enter one pattern (it works like an Exclude pattern) and the most pages to read, from 1 to 10,000,000. A page over its cap is not crawled, rendered or billed, and the report says how many each cap left out. A page that matches several caps needs room under each one, and the start page is always read. You can add up to 10 caps, one for each pattern. A List of URLs scan ignores them. Pages left out may still exist, so a scan with caps never marks pages as gone in Site Pages.

Removing parameters

The Remove parameters field takes names of query parameters, such as utm_*, gclid, fbclid or sessionid, separated by commas or lines. Before PerfBee decides whether it has already seen a page, it takes those parameters off the address, so /shoes?id=7&utm_source=mail and /shoes?id=7&utm_source=ads are one page and are read once. A * matches any characters, names are compared without regard to case, and a name must match whole: id never removes pid. Nothing else in the address changes, so /shoes?id=7 and /shoes?id=8 stay two pages. This is useful with Ignore query parameters switched off, when you want every other parameter to count. With Ignore query parameters on, all parameters except pagination ones are already ignored.

Scanning Specific URLs

Instead of crawling your entire site, you might want to focus on a specific set of pages. Choose List of URLs at the top of the scan form and enter the URLs to check, one per line, on the selected site. The crawler reads only the URLs you specify, and the links on those pages are checked, not followed, unless you choose Follow links from the list: Do not follow links (the default), 1 level, 2 levels or 3 levels. With a level chosen, the crawler also reads the pages of your site that the listed pages link to, that many clicks from the list, and bills them like any page. Links further out are checked only. Max pages and Max depth still stop the scan and Exclude URLs still applies; Include URLs and Page caps do not apply to a list. A line with *, such as /products/*, takes every matching page from your sitemap. A list can hold up to 2,000 URLs. Longer lists are refused with a message that says how many were given; split them into several scans.

Crawl Scope: Internal vs. External Links

By default, the crawler follows the links within your site and does not check links to other sites. Two checkboxes control this:

  • Internal Links: On by default. The crawler follows and checks the links between your pages. When it is off, only the start page is read, plus the pages in your sitemap when Use sitemap is on.
  • External Links: Off by default. When enabled, the crawler checks each external URL to see that it is reachable and which status code it returns. It does not crawl other sites. Checking external links costs no extra credits: only your own site's pages are billed.

Crawl Speed

PerfBeeBot crawls every site at a polite pace and follows your robots.txt, including Crawl-delay. A scan setting can only slow that down: switch on Be gentler under Limits for about one request a second to your site, for its pages, link checks and files, and one page rendered at a time. A big site then takes much longer. The robots.txt and sitemap reads at the start, and the scripts and styles a rendered page loads, are not slowed, and the setting never makes a scan faster than your site's own pace. A faster lane is not a scan setting: if your firewall lets PerfBeeBot in, PerfBee can switch your site to it on request. See Crawl Speed, Robots.txt and Verified Sites for details.

Ignoring Query Parameters

To avoid crawling duplicate content caused by query parameters (e.g., /page?color=red and /page?color=blue), keep the Ignore query parameters option on. It is on by default. URLs that differ only in their query parameters are treated as a single URL, preventing redundant crawls and saving credits. Pagination parameters such as ?page=2 stay separate.

Sitemap Crawling

Leverage your existing sitemap with the Use sitemap setting, which is on by default. The crawler finds your sitemap through robots.txt or at /sitemap.xml and reads every page it lists, in addition to following links found on your site. This also reaches pages that no other page links to.

Checking Images and CSS Resources

Images, CSS and JS files linked on your pages are checked by default. Each has its own checkbox under Check, so you can turn them off. Videos, iframes and media are off by default; when you turn them on, they are checked with a HEAD request only, so nothing is downloaded or played. These checks cost no extra credits. Ensure all your assets are loading correctly to avoid performance issues.

With Images ticked, a Detailed scan also checks images that a page sets as CSS backgrounds, not only those in <img> tags. They show in the Images tab of the scan report with a CSS background label (the label and its filter appear only when the scan found some), a broken one counts under Broken images, and, when a scan has some, the Images download adds a column that says how each image is used. PerfBee finds them in the inline styles of the page, in the page's own <style> rules, and, on a page the scan rendered, in the styles the browser applied. It does not download stylesheet files, so a background set only in a stylesheet is invisible on a page that was not rendered. It checks up to 20 backgrounds on a page that are not already images of that page, never sees what appears only on hover or at another screen size, and gives a background no alt text to check. Checking backgrounds costs no extra credits.

JavaScript rendering

JavaScript rendering, above Advanced settings, is Off by default and pages are read as plain HTML. Auto renders only the page groups that need it, after PerfBee compares a few pages of each group, and Every page renders every page, which is the slowest and most expensive choice. The scan form shows what a rendered page costs. With Auto or Every page chosen, three more options appear:

  • Device: Desktop by default, or Mobile, a 412 px wide touch screen. Its user agent says "Mobile" and still names PerfBeeBot, so a site that tells phones apart by "Android" or "iPhone" will not switch.
  • Extra wait: how long to wait after each page has loaded, for content that appears late: None, 2 s, 5 s or 10 s.
  • Wait for element: a CSS selector such as #price, up to 200 characters. PerfBee reads the page as soon as the element is there. If it is not there when the Extra wait ends, the page is read as it is and counted in the report. The element needs a time limit, so typing one with no Extra wait chosen sets 5 s. Use a plain CSS selector: text=, xpath= and >> chains are not accepted.

A wait makes the scan slower, never more expensive.

Extras: custom extraction and custom search

Under Extras you can add rules that read a value from every page, with a CSS selector, an XPath or a regular expression, and searches that list the pages that contain a phrase, or are missing it. Each has a Test on a page button. They are on every plan and cost no extra credits. See Custom Extraction and Custom Search.

Extras: PageSpeed on a sample of pages

Under Extras, PageSpeed on a sample of pages tests a few of the crawl's pages with PageSpeed once the crawl has ended. It is off until you switch it on, and only for a site in My Sites: for any other address the box is disabled and says so. Set Pages to test from 1 to 30 (10 by default).

  • Which pages: one example page of each of the site's biggest page templates, so the sample is never larger than the crawl and can be smaller than the number you set. When the site has no template with a usable example, PerfBee uses the start page and a page of each kind it knows. A page known to be gone, broken or not an HTML page is left out.
  • What it costs: the credits of one performance test for each page tested, the same on every plan (2 credits). Both devices of a page count as one test. The scan form shows the number of pages and the most it can cost before you start. A page PageSpeed could not test costs nothing, and when your credits run out the rest of the sample is skipped.
  • Where the results are: the Summary tab of the crawl lists each page with its mobile and desktop score, LCP and CLS and a link to the page in Site Pages. Each test also joins that page's history in Page Detail.

PageSpeed is run by Google from its own servers, so a page it cannot reach is not tested. A scheduled scan takes this option from the preset it uses.

Ignore robots.txt (your verified site)

On every plan, a site you have verified in My Sites shows an Ignore robots.txt option in the scan form. See Verifying Your Site.

By using these advanced settings, you can fine-tune your Site Crawler scans to be more efficient, accurate, and tailored to your specific SEO and performance needs. Remember to monitor your credit usage, especially when choosing options that consume more credits, like a Detailed scan or JavaScript rendering. The Credit estimate next to the scan form shows about what a scan will cost before you start.