Skip to content

Site Crawler

Every page, every link, and what changed since the previous crawl.

PerfBeeBot checks every page and link it reaches, and Fix Planner ranks what to fix first. A detailed crawl adds indexing, content and AI readiness checks.

More than broken links.

A fast crawl checks status codes, redirects and links. A detailed crawl adds indexing, content and asset checks on every page.

Every issue, with the pages that have it.

  • Structured data for 15 feature types, hreflang and near-duplicate pages.
  • A link PerfBeeBot could not get a real answer from is Not checked, never broken.
  • Pull any value from every page with a CSS selector, an XPath or a regex.
  • Find the pages that contain a phrase, or the ones missing it.
JavaScript SEO

The HTML your server sends, next to the rendered page.

Rendering is off unless you turn it on. Auto renders 5 sample pages at each depth and renders the rest only where JavaScript adds at least 3 links.

  • Title, meta description, canonical, H1 and robots compared before and after JavaScript.
  • Links that appear only after JavaScript, and the script errors of each page.
  • Desktop or Mobile, an extra wait of up to 10 seconds, or wait for an element.
Search Console

Clicks and index status for the pages you crawl.

Connect Google Search Console read-only. Site Pages then shows what Google reports for each page PerfBee crawled.

  • Clicks and impressions of the last 28 days, and the change against the 28 days before.
  • The numbers and Inspect now cost no credits. Scheduled index checks cover up to 1,000 pages a day for each site.
  • Fix Planner ranks a fix higher when it affects pages with more clicks, and adds two groups: pages with search traffic that are noindex, canonicalised or broken, and indexable pages Google has not indexed.
  • Google Analytics works the same way, read-only: sessions, key events and the sessions from AI assistants, as GA4 reports them.

PerfBee never changes anything in your Google account. It reads search queries only if the workspace owner agrees to it. The numbers stay in your dashboard and your own downloads. PerfBee does not sell them or use them for advertising.

AI Readiness

Can ChatGPT, Claude and Perplexity read your site?

After each completed detailed crawl, PerfBee checks what AI search tools can reach and quote. On every plan, with no extra credits.

  • robots.txt read for each of 14 AI agents, from the search bots to the training bots.
  • Pages kept out of Google’s AI Overviews and AI Mode by noindex, nosnippet or max-snippet:0.
  • Text that appears only after JavaScript runs, which many AI crawlers never see.
  • When an AI search bot loses access, the next detailed crawl tells you: in the app and by email on every plan.

It checks whether AI tools can reach and read your pages. It does not track how often they mention you.

API and MCP

Claude, ChatGPT and your own scripts.

Connect Claude or ChatGPT once and ask what is broken on your site. The tools list sites, start a crawl, follow it, read issues with their HTML element, AI readiness, fixes and what changed, and check whether a fix worked.

  • A REST API and signed webhooks. Reading costs no credits.
  • A crawl you start from code spends credits like one you start in the dashboard.
  • Live links that keep a Google Sheet up to date.

Not in the API, the assistant tools, webhooks or live links: Google numbers, accessibility results and segments.

What Site Crawler checks and does.

Every Site Crawler feature, what it gives you and the plans it is on
FeatureWhat you getPlan
Every crawl, fast or detailed
Pages and linksEvery page and link PerfBeeBot reaches from your sitemap and internal links. It can also check external links, images, CSS, JS, video, iframe and audio sources.Every plan
Broken links and redirectsStatus codes, broken links with the pages that link to them, and redirect chains.Every plan
Blocked URLsMarked Not checked, with the reason: a bot wall, 401, 403, 429, robots.txt or a timeout. Never counted as broken and never billed.Every plan
Big sitesNo page limit: a crawl runs up to 6 days, in 6-hour parts. Start in one folder, exclude URL patterns, cap the pages of a pattern or check a list of up to 2,000 URLs.Every plan
Staging sitesA user name and password or a request header, sent only to a site you verified. No form logins.Every plan
Added by a detailed crawl
IndexabilityNoindex, missing canonicals and canonical chains on every page.Every plan
Titles, descriptions and headingsMissing meta descriptions, missing or multiple H1s, thin content and missing Open Graph tags.Every plan
Structured dataJSON-LD, Microdata and RDFa-Lite compared with the properties Google documents for 15 types of markup.Every plan
HreflangRead from each page and from your XML sitemap: invalid values, missing return links and broken alternates.Every plan
Near-duplicate pagesPages with almost the same text.Every plan
Security headers and mixed contentThe security headers of each page, and files loaded over HTTP on an HTTPS page.Every plan
AI ReadinessWhich of 14 AI agents your robots.txt lets in, pages kept out of Google’s AI Overviews by noindex or nosnippet, and text that appears only after JavaScript. It does not track how often AI tools mention you.Every plan
When you turn it on
JavaScript renderingOff, Auto or Every page. Title, meta description, canonical, H1 and robots compared before and after JavaScript, plus links that appear only after it and script errors.Every plan
Custom extraction and searchA value from every page by CSS selector, XPath or regex, and the pages that contain or miss a phrase.Every plan
AccessibilityOne example page of each of your biggest templates, up to 10 by default, tested against automated rules. You confirm the credits first.Every plan
AI promptsYour instruction, run on the pages you select with your own OpenAI, Anthropic or Google Gemini key. No PerfBee credits.Business and up
Your results
Fix PlannerIssues grouped by page template, ranked by severity, pages affected and links, each with why it ranks there. Check if fixed rechecks only the pages of one fix.Every plan
Site PagesEvery URL with its status, indexability and issues. Segments group URLs by rules, and Compare shows what differs between two pages.Every plan
StructureEach folder with its pages, errors, broken links and click depth, as a tree, a depth chart and a treemap.Every plan
SEO monitoringScheduled crawls, each compared with the one before: new or fixed broken links, and after a detailed crawl, new, gone and changed pages. As often as daily on every plan.Every plan
AlertsWhen a scheduled crawl finishes, finds broken links or fails: in the app and by signed webhook, and by email on Every plan.Every plan
Redirects and sitemapA suggested new address for every page that answers 404 or 410, and a clean sitemap.xml.Every plan
Third-party scriptsWhat each service downloads and how long it keeps the main thread busy, from your performance tests.Every plan
From Google, read-only
Search ConsoleClicks, impressions, average position and index status of each page, for the last 28 days. Search queries only if the workspace owner agrees. Scheduled index checks: Every plan.Every plan
Google AnalyticsSessions, key events and sessions from AI assistants for each page, as GA4 reports them.Every plan
Your data out
ExportsCSV, Excel and PDF of the crawl’s tabs.Every plan
Your brandPDF and Excel carry your logo, name and colour; CSV file names start with your name; share links show your branding.Starter and up
Live linksA Google Sheet that stays up to date with your newest crawl.Every plan
Share linksA public link to a finished crawl, valid for 7, 30 or 90 days.Every plan
API and MCPA REST API, signed webhooks and an MCP server for Claude and ChatGPT. Reading costs no credits.Every plan

Site Crawler, answered.

Anything else? Talk to us.

What does the crawler check?

PerfBeeBot starts from your sitemap, follows internal links and checks every page and link it finds. It can also check external links, images, CSS, JS, video, iframe and audio sources. A detailed crawl adds each page's titles, descriptions, headings, security headers, structured data, hreflang and near-duplicate text, plus the AI Readiness report.

Should I run a fast crawl or a detailed crawl?

Fast checks links and status codes. Detailed adds indexing, content and asset findings for every page, plus the AI Readiness report.

Does it check hreflang, structured data and near-duplicate pages?

Yes, in a detailed crawl. Hreflang is read from each page and from your XML sitemap, and invalid values, missing return links and broken alternates are listed. Structured data (JSON-LD, Microdata, RDFa-Lite) is compared with the properties Google documents for 15 types of markup. Pages with almost the same text are listed as near-duplicates. This compares your markup with Google's documentation; it does not say whether Google will show a rich result.

What happens when a site blocks PerfBeeBot?

The URL is marked Not checked, with the reason: a bot wall, 401, 403, 429, 430, 503, 999, robots.txt, a timeout or a dropped connection. It is never counted as broken and never billed. PerfBee does not get past bot walls; for your own site there are allowlist guides for Cloudflare, Shopify, Akamai, AWS WAF, Vercel and others.

Can it pull custom data or find text on every page?

Yes. Custom extraction reads a value from each page with a CSS selector, an XPath or a regex. Custom search lists the pages that contain a phrase or a regex, or the pages missing it. Test a rule on one page first. Each has its own report tab and CSV or Excel export.

What happens to the issues a crawl finds?

Every finding becomes a fix in Fix Planner: one fix for a page template, the whole site, or the separate pages that have the issue. A broken link is one fix, with the pages that link to it. Fixes are ranked by severity, pages affected, how well linked they are and whether one change fixes them all, and each fix says why it ranks there. With Search Console linked, a fix that affects pages with more clicks ranks higher. Made the change? Click Check if fixed. A crawl reads the pages of that fix again, and only those pages are billed.

Can it monitor my SEO over time?

Yes. For SEO monitoring, schedule a crawl, weekly or daily, and each crawl is compared with the previous one in the same mode. The Changes tab lists pages that are new, gone or changed in status code, title, meta description, indexability or canonical, and broken links that are new or fixed. A fast crawl compares broken links only. A changed title or canonical shows in the Changes tab and is not sent as an alert. A scheduled crawl tells you when it finishes, finds broken links or fails: in the app, by email and by signed webhook.

Can I see Google Analytics numbers next to my pages?

Yes, on every plan, with no credits. Connect Google Analytics read-only and link a Google Analytics 4 property to a site in My Sites. Site Pages then shows each page's sessions and key events for the last 28 days, the change against the 28 days before, and how many sessions came from AI assistants as GA4 reports them. The Conversion tab uses the revenue Analytics reports for a landing page. PerfBee does not read users or visitor identifiers, and it never writes to Analytics.

Can I see Search Console numbers next to my pages?

Yes, with no credits. Connect Google Search Console read-only and link a property to a site in My Sites. Site Pages then shows each page's clicks, impressions and average position for the last 28 days, and the change in clicks against the 28 days before. It also shows Google's index status. Inspect now gives 10 index answers a day for each site. PerfBee reads search queries only if the workspace owner agrees to it. The numbers stay in your dashboard and in your own downloads.

Can it write my redirects?

Yes. The Redirects tab lists every page that answers 404 or 410, most linked first, with a suggested new address: the same page name in another folder, a close name that keeps the model code, or the nearest parent folder, never the homepage. Change or skip any row. Download them for Apache, Nginx, Cloudflare Bulk Redirects or Shopify, or as CSV and Excel. Check published redirects then reads each old URL once, billed like a fast crawl.

Can it make my sitemap?

Yes. The Sitemap tab builds a sitemap.xml from the pages that work and may be indexed, and lists what to remove from the sitemap you publish today, such as pages that answer 404, redirect or say noindex. It writes no lastmod, because PerfBee does not know when a page last changed. Download it as a file.

Can I see my site as folders?

Yes, with no credits. The Structure tab shows each folder with its pages, errors, broken links and click depth, as a folder tree, a depth chart and a treemap. Errors and warnings are known only for pages a detailed crawl read, and folders without such pages say so. Download the tree as CSV or Excel.

Can I group URLs into segments and compare two pages?

Yes, with no credits. A segment is a named group of URLs made from rules, such as every page of your blog, and Site Pages can show one at a time. Compare with another page reads two pages of your site live and shows what differs: status, title, description, canonical, headings, structured data and words. Neither is in the REST API, the assistant tools or live links.

Can I run my own AI prompts on my pages?

Yes, on Business and up, with your own key from OpenAI, Anthropic or Google Gemini. In Site Pages you select pages, write an instruction and run it. The model sees only what Site Pages stores for each page: the title, meta description, headings, a few technical fields and, if you tick it, the main text of the page. It never sees the images or the links, so review every answer. Your provider bills your account and PerfBee uses no credits. PerfBee never applies an answer to your site.

Can it test accessibility?

Yes, as a sample. Accessibility by template tests one example page of each of your biggest templates, up to 10 pages by default, against automated rules and groups the findings by rule and template. You see the pages and the credits first and confirm before anything is spent. Passing every rule does not make a site accessible.

Where do the third-party script numbers come from?

From your performance tests. Lighthouse loads the page in a real browser, so tags that Google Tag Manager adds after load are counted; a crawl of the HTML cannot see them. The Scripts tab adds up the newest test of each page: how much each service downloads and how long it keeps the main thread busy. The crawl adds scripts your pages still ask for that answer an error.

Does it render JavaScript and check JavaScript SEO?

Yes. Choose Off, Auto or Every page in the scan form; Off, plain HTML, is the default. Auto renders 5 sample pages at each URL depth with and without JavaScript, and renders the rest of that depth only if it finds at least 3 links the HTML lacked. Pages that are empty without JavaScript are always rendered. For rendered pages you can choose Desktop or Mobile, add an extra wait, or wait for an element to appear. The report shows where a rendered page differs from the server HTML, and the JavaScript errors it logged.

Can it crawl a staging site or pages behind a password?

Yes, on a site you verified in My Sites. Add a user name and password, or a request header such as a Vercel protection bypass. PerfBee sends them only to that site, over HTTPS. It does not log in through a form, does not get past bot protection and cannot reach a private network or VPN.

How big a site can it crawl?

There is no page limit. A crawl can run for up to 6 days, in 6-hour parts that pick up where the last one stopped. You can start inside one folder, exclude URL patterns, set a page cap for each URL pattern, such as 200 product pages, or check a pasted or uploaded list of up to 2,000 URLs. Be gentler slows a crawl to about one request a second.

Can I use it from code, a Google Sheet, Claude or ChatGPT?

Yes. The REST API, signed webhooks and the MCP server for Claude and ChatGPT are available. Reading costs no credits; a crawl you start through them spends credits like one in the dashboard. None of them can change schedules, and the API and webhooks only work on sites already in My Sites. Search Console numbers, accessibility results and segments are not in them. Live links keep a Google Sheet up to date with your newest crawl.

Can I send a client a white-label report?

Yes, on Starter and up. Set your name, colour and logo once in Settings, under Branding. A public share link to a finished crawl shows them and opens without an account; it is valid for 7, 30 or 90 days and you can revoke it. PDF and Excel exports carry your branding, and CSV file names start with your name. Impact Reports puts the same branding on a report of what you fixed and the clicks before and after.

Find what broke, before your customers do.

  • No page limit
  • 14 days of Business free
  • No card