Site CrawlerFree tool
Website Crawler
Crawl up to 50 pages of a site and see the SEO issues on each one.
- Up to 50 pages
- 60 seconds
- as PerfBeeBot
- no account
Result
Paste your site address above
PerfBeeBot reads your robots.txt, then follows links on the same host, at most two requests at a time, for up to 60 seconds or 50 pages.
- The status code of each page
- Missing, short or long titles
- Missing or long meta descriptions
- Missing H1
- Noindex in the robots meta tag
- Pages slower than 3 seconds
What this check covers, and what it doesn’t.
A quick look at up to 50 pages. For a whole site, see below.
It checks
- The status code of each pageYes
- Missing, short or long titlesYes
- Missing or long meta descriptionsYes
- Missing H1Yes
- Noindex in the robots meta tagYes
- Pages slower than 3 secondsYes
It doesn’t
- Pages after the first 50No
- Subdomains and other sitesNo
- Pages linked only from JavaScriptNo
- Your sitemapNo
- Pages robots.txt keeps PerfBeeBot fromNo
- Crawls on a scheduleNo
Small problems repeat across pages.
A missing description on one template is missing on every page built from it. A short crawl shows which issues repeat, so you can fix the template once.
How to use it
Paste your site address
The crawl starts there and stays on that host.
Select Start crawl
PerfBeeBot reads your robots.txt, then follows links on the same host, at most two requests at a time, for up to 60 seconds or 50 pages.
Fix the repeat issues first
An issue on many pages usually lives in a shared template. Fix it there once.
Check every page with Site Crawler
Crawl your whole site on a schedule, as often as daily on every plan.
- This free toolUp to 50 pages · 60 seconds, on demand
- Site CrawlerYour whole site, on a schedule
Is the Website Crawler free?
Yes. You can run it without an account. The result shows the first 5 pages; enter your email to see every page it crawled.
How does it pick pages?
It starts at the URL you enter and follows links in the HTML, on the same host, in the order it finds them. It skips images, CSS, JavaScript and PDF files.
Why did it stop before 50 pages?
The free crawl stops after 60 seconds, or when it runs out of links to follow. A site that asks for a Crawl-delay in robots.txt gets fewer pages, because PerfBeeBot waits between requests as asked.
Does it follow robots.txt?
Yes. It reads robots.txt first and skips the pages it asks PerfBeeBot not to crawl. If robots.txt does not answer, it does not crawl at all.
Does it run JavaScript?
No. It reads the HTML the server sends to PerfBeeBot. Anything added later by JavaScript is not seen. Site Crawler in the dashboard can render pages with JavaScript.
How often can I run it?
Up to 3 crawls an hour. Website Crawler and Duplicate Meta Tag Checker share that limit. When you reach the limit, the page tells you how long to wait.
More free checks.
- Meta Tag CheckerTitle, description, canonical, robots, Open Graph and viewport.One page
- HTTP Headers CheckerOne page’s response headers, with security and cache headers.One URL
- SSL Certificate CheckerCertificate, expiry and HTTPS setup of one domain.One domain
- Robots.txt TesterWhat robots.txt allows and blocks for Googlebot.One robots.txt · Googlebot