Site Crawler
Finds broken links, redirect chains, missing meta tags and indexability problems across every page it can reach.
Know what broke and what went missing, before your customers do.
Crawl every page. Catch SEO issues, slow templates and missing content.
- Full crawl in minutes
- No credit card
- Nothing to install
Illustration: PerfBee's crawler walks the page tree of catalog.example, flags a user-manual link returning 404, measures performance at 62 on mobile and 91 on desktop, then finds the Documents tab missing on a product page and raises a drift alert for 240 pages.
The platform
One platform for every page you can't check by hand.
Start with a crawl of the whole site. Add lab tests, real-visitor data, and presence checks as your catalog and team grow.
Performance Monitoring
Lighthouse tests on mobile and desktop, with Core Web Vitals tracked over time.
Real User Monitoring
Field data from your actual visitors, next to your lab scores.
Content Monitor
Checks that required elements like energy labels, manuals and document tabs are still on every page after each template change.
Site Crawler
Every page. Every status code. Every missing tag.
A cloud crawl of your whole site, not a single-URL test. Start it from the browser and get a full inventory of what's broken, redirected or hidden from search.
- Pages crawled
- 12,480
- Crawl mode
- Detailed
- Issues found
- 647
- Finished
- 03:42
| URL | Status | Depth | Indexable | Title | Meta desc. | Finding |
|---|---|---|---|---|---|---|
| /products/washer-wm-8120 | 200 | 2 | ✓ Yes | ✓ | ✓ | No issues |
| /support/manual-wm-8120.pdf | 404 | 3 | n/a | n/a | n/a | ✕ Broken link · 214 inlinks |
| /category/dishwashers | 301301200 | 1 | ✓ Yes | ✓ | ✓ | ▲ Redirect chain · 2 hops |
| /products/oven-bo-5520 | 200 | 2 | ✓ Yes | ✓ | ✕ Missing | ▲ Missing meta description |
| /products/fridge-rf-9100 | 200 | 2 | ● noindex | ✓ | ✓ | ● Non-indexable · meta robots |
| /blog/energy-labels-explained | 200 | 3 | ✓ Yes | ▲ Duplicate | ✓ | ▲ Duplicate title · 3 pages |
| /products/hood-ch-9020?color=black | 200 | 3 | ● Canonical | ✓ | ✓ | ● Canonicalised to /products/hood-ch-9020 |
| /campaign/spring-2025 | 410 | 2 | n/a | n/a | n/a | ✕ Broken link · in main navigation |
A crawler that plays by the rules
PerfBeeBot identifies itself in every request and follows your robots.txt. It does not bypass bot protection, firewalls or access restrictions. If a page is blocked to bots, the report says so.
Pay per page crawled
- Fast crawl
- 1 credit / page
- Detailed crawl
- 4 credits / page
A detailed crawl of 12,480 pages uses 49,920 credits, inside Business's 50,000 a month.
Allowed in, blocked out
PerfBeeBot reads your robots.txt first, crawls what it allows and skips the rest.
Performance · AI Search Readiness
Measure the speed. Check who's allowed in.
Lighthouse tests show how fast your key templates are. AI Search Readiness shows whether AI crawlers can reach and read them.
/products/washer-wm-8120
Performance Monitoring
Lighthouse score over the last 8 weekly tests: desktop rises from 84 to 91, mobile from 48 to 62. Mobile LCP 3.1 seconds needs work, CLS 0.04 is good, INP 240 milliseconds needs work.
Mobile and desktop Lighthouse runs, tracked over time. INP is a field metric and appears when real-user data is available for the URL.
catalog.example/robots.txtNo extra credits
AI Search Readiness
| AI user-agent | robots.txt rule | Access |
|---|---|---|
| GPTBot | Allow: / | ✓ Allowed |
| ClaudeBot | Allow: / | ✓ Allowed |
| PerplexityBot | Disallow: / | ✕ Blocked |
| Google-Extended | Disallow: / | ✕ Blocked |
| CCBot | Disallow: /products/ | ✕ Blocked |
▲Content needs JavaScript to appear37 pages
✕Structured data problems112 pages
Runs after each detailed crawl, on every plan. It shows what AI crawlers can access and read. Passing these checks does not guarantee visibility or rankings in AI search.
Real User MonitoringBusiness and up
Lab scores show what could happen. Real visitors show what did.
See LCP, INP and CLS from the phones, browsers and networks your customers actually use.
<script src="https://cdn.perfbee.com/rum.js"
data-token="YOUR_SITE_TOKEN" async></script>
One script tag. Under 2 KB gzipped.
Phones, laptops and tablets send real page-view data to PerfBee. Sample field data for the last 28 days: p75 LCP 2.9 seconds on mobile, needs work; 1.8 seconds on desktop, good; INP 210 milliseconds, needs work; CLS 0.06, good.
Content MonitorEnterprise
A crawler sees a working page. Content Monitor sees what's no longer on it.
Every page can return 200 and still be missing the energy label, the manual or the documents tab your customers need. Content Monitor checks for them, page by page.
- 01 · Tuesday 17:20A template change ships
A product-page update removes the documents tab component.
- 02 · Wednesday 04:00The scheduled scan runs
Content Monitor checks every monitored page for each required element.
- 03 · Wednesday 04:06Your team gets a drift alert
Listing the pages that lost it, before customers start asking.
1,860 product pages monitored
Coverage by required element
Energy label1,852 / 1,860▲ 8 missing
User manual1,841 / 1,860▲ 19 missing
Documents tab1,620 / 1,860✕ 240 missing · new
Banner1,860 / 1,860✓ All present
Structured data1,794 / 1,860▲ 66 missing
| Sample pages | Energy label | User manual | Documents tab | Banner | Structured data |
|---|---|---|---|---|---|
| /products/washer-wm-8120 | Present | Present | Missing | Present | Present |
| /products/washer-wm-9140 | Present | Present | Missing | Present | Present |
| /products/dryer-td-7300 | Present | Present | Present | Present | Present |
| /products/dishwasher-dw-6040 | Present | Present | Missing | Present | Present |
| /products/oven-bo-5520 | Present | Present | Missing | Present | Present |
| /products/fridge-rf-9100 | Present | Present | Missing | Present | Present |
| /products/hob-ih-6400 | Present | Present | Present | Present | Missing |
| /products/hood-ch-9020 | Present | Missing | Missing | Present | Present |
Drift detectedDocuments tab missing240affected pages of 1,860 monitored · first seen in the 04:00 scan
Documents tab coverage · vs the previous scan
- Previous scan · Oct 12, 04:00
- 100%
- This scan · Today 04:00
- 87%
Affected pages
- /products/washer-wm-8120
- /products/washer-wm-9140
- /products/dishwasher-dw-6040
- + 237 more pages
Presence, not proofreading. It checks that each required element exists on the page, not whether the text inside it is factually correct.
Runs on your schedule. Checks run with your scheduled scans and alert the team when something drifts.
Built for big catalogs. Manufacturers and retailers with thousands of product pages that share a few templates.
Also in PerfBee
From finding the problem to shipping the fix.
Suggestions your developers can review, benchmarks against competitors, and reports your clients can read.
AI Code Fix Suggestions
Business and upEach Lighthouse finding comes with a proposed change. Your team reviews it and decides. PerfBee never edits your site.
▲Largest Contentful Paint image was lazily loaded
/products/washer-wm-8120 · mobile
/products/washer-wm-8120 · mobile
−<img src="/media/wm-8120.webp" loading="lazy">
+<img src="/media/wm-8120.webp" fetchpriority="high"
+ width="1200" height="900">
Competitor Analysis
Core Web Vitals side by side, mobile and desktop.
| Site | LCP mob. | LCP desk. |
|---|---|---|
| Your page | 3.1 s | 1.1 s |
| Competitor A | 2.6 s | 1.6 s |
| Competitor B | 3.6 s | 2.1 s |
Critical CSS
Extract the above-the-fold CSS for a page, ready to inline.
.hero{min-height:520px}
.hero h1{font:800 3rem/1.05 var(--head)}
.buy{background:#ffc71f;border-radius:10px}
Read about Critical CSS Scheduled scans & alerts
Detailed crawlWeekly · Mon 03:00
Performance testsDaily · 06:00
Scan complete · catalog.example
3 new broken links since last scan
3 new broken links since last scan
Weekly scans on Free and Starter. Daily scans from Business.
Team workspaces
Business and up- AdminBilling, members
- MemberRun scans
- ViewerRead reports
- Business
- 3
- Business+
- 10
- Enterprise
- ∞
Seats per plan.
For agencies
Your logo. Your colors. Their site's issues, explained.
Manage every client site in one workspace and send white-label PDF reports that look like they came from your team.
Your agency logo
Technical SEO & Performance · September 2026Site health report
catalog.example
- Pages crawled
- 12,480
- Broken links
- 38
- Mobile score
- 62
- Redirect chains
- 112
- Missing meta descriptions
- 406
- Non-indexable pages
- 91
Prepared by [Your agency] · Confidential1
Workspace · Client site A
- [email protected]YouOwner
- [email protected]TeamMember
- [email protected]ClientViewer
Export what the client needs
CSVExcelPDF · white-label
CSV, Excel and PDF exports from Starter upwards. White-label reports carry your agency's logo and brand color. Give clients viewer access so they can follow progress without changing settings.
How PerfBee compares
Good tools, different jobs.
GTmetrix, Ahrefs and Screaming Frog are excellent at what they focus on. Here's where PerfBee's scope differs, based on each vendor's public documentation.
Compare PerfBee with
| Capability | PerfBee | GTmetrix | Ahrefs Site Audit | Screaming Frog |
|---|---|---|---|---|
| Runs in the cloud, nothing to install | Yes | Yes | Yes | Desktop app for Windows, macOS and Linux |
| Site-wide technical SEO crawl | Yes, every plan | Not listed, page-level testing | Yes | Yes · free tier up to 500 URLs |
| Scheduled scans | Weekly; daily from Business | Page monitoring, monthly to hourly (hourly on top plans) | Yes | Paid licence; runs on your machine |
| Lighthouse lab tests | Mobile and desktop | Yes | Via PageSpeed Insights API with your own Google key | Via PageSpeed Insights integration, paid licence |
| Real-user field data | Own RUM script, Business and up | Chrome UX Report data, where available | Chrome UX Report data via PageSpeed API (your own key) | Chrome UX Report data via PageSpeed API, paid licence |
| Critical CSS extraction | Yes, above-the-fold CSS per page | Not listed. Flags render-blocking CSS in Lighthouse results | Not listed | Not listed. Reports render-blocking and unused CSS via PageSpeed integration |
| Alerts when required page elements go missing | Content Monitor, Enterprise | Not listed | Custom issues with email alerts (e.g. an 'Out of stock' label) | Paid licence: 'does not contain' custom search; no element alerts listed |
Based on vendors' public documentation, checked October 2026. “Not listed” means we didn't find it documented, not that it can't be done. Product names belong to their owners.
Pricing
Pay for results, not seat licenses.
Start free. Upgrade when your catalog needs more monitoring.
| Compare plans | |||||
|---|---|---|---|---|---|
| Included on every plan | |||||
| Site Crawler · broken links | Included | Included | Included | Included | Included |
| Performance analysis | Included | Included | Included | Included | Included |
| AI Search Readiness report | Included | Included | Included | Included | Included |
| Critical CSS extraction | Included | Included | Included | Included | Included |
| Unlimited sites | Included | Included | Included | Included | Included |
| Data & reporting | |||||
| Data retention | 7 days | 14 days | 30 days | 90 days | 120 days |
| CSV, Excel & PDF export | Not included | Included | Included | Included | Included |
| Email notifications | Not included | Included | Included | Included | Included |
| Monitoring | |||||
| Scheduled scans | 1 weekly | 2 weekly | 10 daily | 50 daily | Unlimited |
| Real User Monitoring | Not included | Not included | Included | Included | Included |
| AI insights | Not included | Not included | Included | Included | Included |
| Content Monitor · drift alerts | Not included | Not included | Not included | Not included | Included |
| Team & support | |||||
| Team members | Not included | Not included | 3 | 10 | Unlimited |
| Priority queue | Not included | Not included | Not included | Included | Included |
| Priority support & SLA | Not included | Not included | Not included | Not included | Included |
Credits reset monthlyNo contracts, cancel anytime7-day money-back guaranteePrices in USD. Depending on your country, tax is included in the price or added at checkout.
What does PerfBee crawl?
Starting from your site's address, PerfBeeBot follows internal links and checks every page it can reach: status codes, broken links, redirect chains, meta tags, indexability and other technical SEO signals. It identifies itself and follows your robots.txt.
What's the difference between the free audit and a full crawl?
The free audit tests one URL with Google PageSpeed and returns mobile and desktop scores. A full crawl checks every page across your site for technical issues and uses credits from your plan.
How do credits work?
Each plan includes a monthly credit allowance. A fast crawl uses 1 credit per page, a detailed crawl 4 credits per page, and a performance test 2 credits. AI Search Readiness runs after detailed crawls at no extra cost.
What is AI Search Readiness?
A check that runs after each detailed crawl, on every plan. It shows which AI crawlers your robots.txt allows or blocks, which pages need JavaScript to reveal their content, and where structured data has problems. It doesn't guarantee visibility or rankings in AI search.
How does Content Monitor detect missing elements?
You choose the elements that must be present on a set of pages, an energy label, a user manual, a documents tab. Each scheduled scan checks every monitored page for them and sends a drift alert listing the affected pages when one disappears. It checks presence, not the accuracy of the text. Available on Enterprise.
Does PerfBee change my code automatically?
No. AI Code Fix Suggestions propose changes for your team to review and apply. PerfBee never edits your website.
Which plans include Real User Monitoring?
Business, Business+ and Enterprise. It needs one script tag on your site, under 2 KB gzipped.
Can agencies export white-label reports?
Yes. CSV, Excel and PDF exports are available from Starter upwards, and white-label PDF reports use your agency's own logo and brand color. The Free plan doesn't include exports.
Find the problems hiding across your site.
Start with a free audit. Move to a full crawl when you're ready.