Indexing & CrawlabilityFree tool
Robots.txt Tester
Read a site’s robots.txt and test whether Googlebot may fetch a path.
- One robots.txt
- Googlebot
- as PerfBeeBot
- no account
Result
Enter a domain above
PerfBeeBot fetches /robots.txt, reads it the way RFC 9309 describes and tests your path for Googlebot.
- Whether robots.txt exists, and its status code
- The rules for each user agent
- Sitemap lines
- Whether Googlebot may fetch one path
What this check covers, and what it doesn’t.
A quick look at one robots.txt. For a whole site, see below.
It checks
- Whether robots.txt exists, and its status codeYes
- The rules for each user agentYes
- Sitemap linesYes
- Whether Googlebot may fetch one pathYes
It doesn’t
- Bots other than GooglebotNo
- noindex tags on the pageNo
- Whether the page is in Google’s indexNo
- Firewall or CDN blocksNo
One line can hide a whole site.
robots.txt tells crawlers where they may go. A “Disallow: /” left over from a staging site keeps search engines out of everything.
How to use it
Enter a domain
With or without https://. No account needed.
Select Test robots.txt
PerfBeeBot fetches /robots.txt, reads it the way RFC 9309 describes and tests your path for Googlebot.
Fix the rule, not the symptom
If a page you want in search is blocked, change the Disallow line that matches it. robots.txt does not remove pages from the index; noindex does.
Check every page with Site Crawler
Crawl your whole site on a schedule, as often as daily on every plan.
- This free toolOne robots.txt · Googlebot, on demand
- Site CrawlerYour whole site, on a schedule
Is the Robots.txt Tester free?
Yes. You can run it without an account or an email address.
Does blocking a page in robots.txt keep it out of Google?
Not always. Google can still index a blocked URL that other pages link to, without reading its content. To keep a page out, allow crawling and use noindex.
How are the rules read?
As RFC 9309 and Google describe: a group that names Googlebot replaces the * group, the longest matching rule wins, and Allow wins a tie.
What if robots.txt returns an error?
A 404 or another 4xx means there are no rules, so everything is allowed. A 5xx error means crawlers treat the whole site as blocked until robots.txt loads again.
How often can I run it?
Up to 20 tests an hour. When you reach the limit, the page tells you how long to wait.
More free checks.
- XML Sitemap ValidatorYour sitemap and its first 20 URLs.One sitemap · first 20 URLs
- Google Index CheckerWhether anything stops Google from indexing a page.One page
- Noindex Tag CheckerWhether a page asks not to be indexed.One page
- Website Speed TestOne Lighthouse test on mobile, with the top fixes.One page · mobile