Skip to content

Indexing & CrawlabilityFree tool

Robots.txt Tester

Read a site’s robots.txt and test whether Googlebot may fetch a path.

  • One robots.txt
  • Googlebot
  • as PerfBeeBot
  • no account

Result

Resultnothing checked yet

Enter a domain above

PerfBeeBot fetches /robots.txt, reads it the way RFC 9309 describes and tests your path for Googlebot.

  • Whether robots.txt exists, and its status code
  • The rules for each user agent
  • Sitemap lines
  • Whether Googlebot may fetch one path

What it checks

What this check covers, and what it doesn’t.

A quick look at one robots.txt. For a whole site, see below.

It checks

  • Whether robots.txt exists, and its status codeYes
  • The rules for each user agentYes
  • Sitemap linesYes
  • Whether Googlebot may fetch one pathYes

It doesn’t

  • Bots other than GooglebotNo
  • noindex tags on the pageNo
  • Whether the page is in Google’s indexNo
  • Firewall or CDN blocksNo
Why it matters

One line can hide a whole site.

robots.txt tells crawlers where they may go. A “Disallow: /” left over from a staging site keeps search engines out of everything.

How to use it

  1. Enter a domain

    With or without https://. No account needed.

  2. Select Test robots.txt

    PerfBeeBot fetches /robots.txt, reads it the way RFC 9309 describes and tests your path for Googlebot.

  3. Fix the rule, not the symptom

    If a page you want in search is blocked, change the Disallow line that matches it. robots.txt does not remove pages from the index; noindex does.

Every plan

Check every page with Site Crawler

Crawl your whole site on a schedule, as often as daily on every plan.

  • This free toolOne robots.txt · Googlebot, on demand
  • Site CrawlerYour whole site, on a schedule
FAQ

Robots.txt Tester, answered.

Anything else? Talk to us.

Is the Robots.txt Tester free?

Yes. You can run it without an account or an email address.

Does blocking a page in robots.txt keep it out of Google?

Not always. Google can still index a blocked URL that other pages link to, without reading its content. To keep a page out, allow crawling and use noindex.

How are the rules read?

As RFC 9309 and Google describe: a group that names Googlebot replaces the * group, the longest matching rule wins, and Allow wins a tie.

What if robots.txt returns an error?

A 404 or another 4xx means there are no rules, so everything is allowed. A 5xx error means crawlers treat the whole site as blocked until robots.txt loads again.

How often can I run it?

Up to 20 tests an hour. When you reach the limit, the page tells you how long to wait.