Custom extraction reads a value from every page a crawl visits, such as a price, a SKU or the number of reviews. Custom search lists the pages that contain a phrase, or the pages that are missing it. Both are on every plan and cost no extra credits.
You set them up in the scan form: open Site Crawler, then Scan, open Advanced settings and go to Extras. A scan can have up to 20 extraction rules and 10 searches. Save them in a preset to use them again, and a scheduled scan made from that preset runs them too.
Custom extraction
Click Add rule, give the rule a Name and choose how to find the value:
- Find with: a CSS selector (
span.price), an XPath (//span[@itemprop='price']) or a Regex in the HTML ("sku":"(\w+)"). - Extract: the element's Text, an Attribute (such as
hrefordata-sku), its Inner HTML, its Outer HTML, or a Count of the elements found. - Matches: All on the page or First only.
- Regex (optional): keep only the part of the value a regular expression matches. Group picks one capture group, for example
1to keep the number in(\d+) TL.
An XPath can also end in an attribute (//a/@href) or be a function such as count(//img).
The values appear in the report's Extraction tab: one row per page, one column per rule, and Not found where a rule found nothing. A rule keeps up to 100 values per page; the count shows how many there were.
Custom search
Click Add search, give it a Name, and set:
- List the pages that: Contain it or Do not contain it.
- Look for: the words to find. Tick It is a regular expression for a pattern, and Match case if capitals matter.
- Where: The text visitors read (without scripts and styles) or The page's HTML.
- Only inside (optional): The whole page, or only inside the elements a CSS selector or an XPath finds, such as
mainor//footer.
Some examples: pages that still say "Out of stock", pages without your analytics tag (search the HTML for its ID), product pages without application/ld+json, or an old phone number left in a footer.
The results appear in the report's Custom search tab, one chip per search. For "contain" searches each page shows how often the text was found, with a short snippet around the first match.
Test on a page first
Each editor has a Test on a page button. Enter a Page to test on the selected site and click Test. PerfBeeBot fetches that page and shows what every rule and search finds. Tick Also rendered to see the result after JavaScript runs too. If the page needs JavaScript for the value, turn on JavaScript rendering for the scan. Testing costs no credits. The page must be on a site in My Sites, and robots.txt must allow it.
Limits
- A regular expression reads the first 500,000 characters of a page.
- A pattern that can take very long to run, such as
(a+)+, is refused when you test it or start the scan, with the reason. - A rule or search that takes more than two seconds on three pages is switched off for the rest of that scan.
Export
Both tabs have an Export menu for CSV and Excel files. Exports are included in Starter and up.