A crawler that keeps running after you close your laptop.

Screaming Frog runs on your machine, on demand, from your IP. PerfBee runs in our infrastructure on a schedule, keeps every crawl so it can show you what changed, and escalates through three different fetch strategies when a site blocks it. The honest trade: Screaming Frog is still the better tool for a deep one-off audit on your desk — it has log-file analysis, GSC and GA integration, JavaScript-render configuration and custom extraction that we do not match. PerfBee is the better tool when the crawl has to keep happening without you.

Crawling is metered per page, not per scan: 1 credit per page for a fast scan and 4 per page for a detailed audit. JS rendering is charged at the detailed rate. Verified sites you own are cheaper. Before you start, PerfBee reads your sitemap and shows the estimated page count, the cost for your plan and whether your remaining credits cover it. The free plan includes 500 credits a month, which is one 500-page fast scan.

PerfBee first requests pages with a client that reproduces a real Chrome TLS fingerprint. If that is refused it retries over plain HTTP, and on a 403, 429 or 503 it escalates that request to a managed headless Chromium. Challenge and interstitial pages are recognised from the response body and reported separately from healthy pages, so a wall of Cloudflare pages does not show up as a wall of 200s.

No, and this is deliberate. Media URLs — video, audio, iframe and embed sources — are checked with a HEAD request only, from a bot user agent, with no Referer and no credentials. The body is never fetched, so no view, play or impression is recorded and nothing reaches your analytics.

No. Cookies are disabled for the whole crawl, so staging environments and member areas that require a session cannot be reached today. Sites protected only at the network level can be crawled if you allowlist us.

Yes, as an option. With JS rendering on, every page is loaded in a headless Chromium and the rendered DOM is what gets parsed, so content injected by a framework is seen. It is significantly slower and charged at the detailed rate, so it is normally used for a targeted crawl rather than a nightly one.

Yes, on every detailed crawl, at no extra cost. robots.txt is evaluated for 14 AI crawlers — OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta and Common Crawl — against every crawled URL, and sample pages are fetched as each crawler next to a normal browser to catch firewall blocks. Every page is checked for noindex, nosnippet and max-snippet in the meta tag or the X-Robots-Tag header, content that only appears after JavaScript runs, and broken or duplicated structured data. It does not tell you whether an AI answer actually cites you, and a block tied to a crawler's verified IP addresses cannot be reproduced from outside.

Crawls can be scheduled daily, weekly or monthly, and each run is diffed against the previous one so the Changes tab is ready when you open it. When a crawl finishes you get an in-app notification, and an email when it found broken links or failed — set per event, on plans that include email notifications. You are also told when an AI search crawler loses access to the site. Emailed summaries are still available as a digest — weekly by default, monthly optional, daily if you turn it on. There is no Slack integration yet.

Yes. Any completed scan can be given a public link that expires after 7, 30 or 90 days, viewable without logging in, and revoked at any time. Results also export to CSV from every tab, filtered exactly as you left them on screen.

Crawls run on our workers with a six-hour ceiling per run and a page cap you set yourself, and tens of thousands of URLs per crawl is normal. For a very large catalogue the practical approach is several scoped schedules — one per section — rather than one crawl of everything.