Skip to content

Site CrawlerFree tool

Website Crawler

Crawl up to 50 pages of a site and see the SEO issues on each one.

  • Up to 50 pages
  • 60 seconds
  • as PerfBeeBot
  • no account

Result

Resultnothing checked yet

Paste your site address above

PerfBeeBot reads your robots.txt, then follows links on the same host, at most two requests at a time, for up to 60 seconds or 50 pages.

  • The status code of each page
  • Missing, short or long titles
  • Missing or long meta descriptions
  • Missing H1
  • Noindex in the robots meta tag
  • Pages slower than 3 seconds

What it checks

What this check covers, and what it doesn’t.

A quick look at up to 50 pages. For a whole site, see below.

It checks

  • The status code of each pageYes
  • Missing, short or long titlesYes
  • Missing or long meta descriptionsYes
  • Missing H1Yes
  • Noindex in the robots meta tagYes
  • Pages slower than 3 secondsYes

It doesn’t

  • Pages after the first 50No
  • Subdomains and other sitesNo
  • Pages linked only from JavaScriptNo
  • Your sitemapNo
  • Pages robots.txt keeps PerfBeeBot fromNo
  • Crawls on a scheduleNo
Why it matters

Small problems repeat across pages.

A missing description on one template is missing on every page built from it. A short crawl shows which issues repeat, so you can fix the template once.

How to use it

  1. Paste your site address

    The crawl starts there and stays on that host.

  2. Select Start crawl

    PerfBeeBot reads your robots.txt, then follows links on the same host, at most two requests at a time, for up to 60 seconds or 50 pages.

  3. Fix the repeat issues first

    An issue on many pages usually lives in a shared template. Fix it there once.

Every plan

Check every page with Site Crawler

Crawl your whole site on a schedule, as often as daily on every plan.

  • This free toolUp to 50 pages · 60 seconds, on demand
  • Site CrawlerYour whole site, on a schedule
FAQ

Website Crawler, answered.

Anything else? Talk to us.

Is the Website Crawler free?

Yes. You can run it without an account. The result shows the first 5 pages; enter your email to see every page it crawled.

How does it pick pages?

It starts at the URL you enter and follows links in the HTML, on the same host, in the order it finds them. It skips images, CSS, JavaScript and PDF files.

Why did it stop before 50 pages?

The free crawl stops after 60 seconds, or when it runs out of links to follow. A site that asks for a Crawl-delay in robots.txt gets fewer pages, because PerfBeeBot waits between requests as asked.

Does it follow robots.txt?

Yes. It reads robots.txt first and skips the pages it asks PerfBeeBot not to crawl. If robots.txt does not answer, it does not crawl at all.

Does it run JavaScript?

No. It reads the HTML the server sends to PerfBeeBot. Anything added later by JavaScript is not seen. Site Crawler in the dashboard can render pages with JavaScript.

How often can I run it?

Up to 3 crawls an hour. Website Crawler and Duplicate Meta Tag Checker share that limit. When you reach the limit, the page tells you how long to wait.