Crawl the live site before the move and keep its URL list, crawl the staging site before launch, and crawl the new site on launch day. Then check every old URL with a List of URLs scan and send the broken ones to new pages with redirects.
- Plans
- Every plan. Staging access is Starter and up, and so are downloads.
- Credits
- 1 a page for a fast crawl, 2 for a detailed one.
- On the Free plan
- Crawls, list scans and the Changes and Redirects tabs.
Before the move, crawl the live site and keep its URLs
Run a detailed crawl of the site as it is now. A detailed crawl reads the title, meta description, canonical and indexability of each page. A fast crawl checks status codes and links only.
Then download the URL list: All URLs in the scan report, or the Site Pages list, on Starter and up. On the Free plan, read it through the REST API. A report is kept only for your plan's data retention period, so keep your own copy.
That list is your redirect plan. Every old URL needs a new address, or a reason to go.
Crawl the staging site behind its password
Add the staging host to My Sites and verify it. Then open Staging access for that site. Give PerfBee a user name and password, or a request header your host asks for. PerfBee sends them only to that host, over HTTPS, and a redirect to another host drops them.
It does not log in through a login form. It cannot reach a site that is only on a private network or behind a VPN. The tunnel article shows how to give such a site a public address that only PerfBee can open.
On staging, look for broken links, redirect chains and pages that cannot be indexed. A staging site is often set to noindex on purpose. Note it, so that you can check the live site does not keep it.
On launch day, crawl the new site in full
Run a detailed crawl the moment the new site is live. The Changes tab of its report compares it with the previous detailed crawl of the same site:
- Pages that are new, gone or changed. A changed page shows what moved: status code, title, meta description, indexability or canonical.
- Broken links that are new, and the ones that are fixed.
A page counts as gone in two cases. The new crawl reached it and it no longer answers as a page, for example with an error or a redirect. Or the new crawl covered at least what the last one did and did not find it. Any other missing page is counted apart as not reached, for example one the crawl was blocked from.
A fast crawl compares broken links only. So does any comparison where a crawl has more than 100,000 pages.
Then open Site Pages and set Indexability to Not indexable. A noindex left over from staging shows there. If robots.txt still shuts crawlers out, the pages it kept PerfBeeBot from show as not checked, with robots.txt as the reason.
Check every old URL with a List of URLs scan
In the scan form, choose List of URLs. Paste the old URLs, or upload them as a .txt or .csv file, up to 2,000 in one scan. Split a longer list into several scans.
PerfBee checks each URL and records its status code and every redirect hop. You see a chain of redirects, a redirect to a page that fails, or an old URL that still answers 404.
Fix old URLs that answer 404 in the Redirects tab
The Redirects tab of a site in My Sites lists the site's own pages that answer 404 or 410. The most linked come first. A URL shows up there once a scan has checked it, so the list scan above fills it. Where one fits, a row gets a suggested new address of one of three kinds:
- The same page name: a live page whose URL ends the same way, for example after the page moved to another folder.
- A similar name: in the same first folder, the live page whose name shares the most words.
- The parent folder: the nearest live folder above the old URL.
When nothing fits, the row waits for an address you type.
The homepage is never suggested. Google treats a redirect to the homepage as a soft 404: the old page's value is lost anyway.
Change or leave out any row. Then download the redirects for Apache, Nginx, Cloudflare Bulk Redirects or Shopify, or as CSV or Excel, on Starter and up. Every redirect is a 301. Nothing changes on your site until you publish the file.
Once the redirects are live, Check published redirects reads up to 2,000 ready old URLs once each and says whether each lands where you set. It is billed like a fast crawl for the URLs it reads, and it shows the credits first. The redirect map article covers every step.
In the weeks after, crawl on a schedule
Schedule a weekly detailed crawl. Every plan has at least one schedule, and each crawl's Changes tab compares it with the one before.
In Site Pages, the Not in latest scan view lists URLs that earlier scans found and the newest one did not. The Links out view lists links to other hosts and to pages outside the crawled folder. Search it for your staging host name to find a link left pointing at staging.
A scheduled crawl tells you when it finds broken links: in the app on every plan, and by email on Starter and up.