GEO audit: can AI crawlers read your site?
PerfBee’s AI search readiness report is a technical GEO audit. It checks what ChatGPT, Claude, Perplexity and Google’s AI features can reach and quote on your site. It runs after each completed detailed crawl, on every plan, with no extra credits.
What the AI readiness check reads
It checks whether AI tools can reach and read your pages. It does not track how often they mention you.
Which AI bots can crawl my site?
PerfBee checks your robots.txt for each of 14 AI agents. A group that names a bot applies to it instead of the * group. For every bot you see whether it may read your homepage and how many of your crawled pages are closed to it.
Which pages can AI answers quote?
A page that is not indexable, or carries nosnippet or max-snippet:0 in its robots meta tag or X-Robots-Tag header, stays out of Google’s AI Overviews and AI Mode. PerfBee lists those pages by issue, and pages with a shorter max-snippet as a warning.
Is my structured data clean?
Duplicate, invalid or incomplete JSON-LD sends AI answers mixed or missing facts. PerfBee lists the pages for each, and checks that your homepage says who you are with Organization markup and sameAs links.
What does my robots.txt say about AI use?
Content-Signal lines state whether your content may be used for search, as AI input or for AI training. PerfBee shows them per user-agent group. They state a preference; they do not block anything.
Do I need llms.txt?
PerfBee checks whether /llms.txt exists. It is optional: Google says no special file is needed to appear in its AI features.
Does my content need JavaScript?
Many AI crawlers read only the HTML your server sends. A page whose HTML is an empty app shell looks blank to them. PerfBee flags those pages, and the report compares two sample pages before and after JavaScript runs.
The screen: one page from a crawl with JavaScript rendering, which compares every page this way (4 credits a rendered page).
The 14 AI bots in the report
AI search bots, page visits a user asked for, and model training. Blocking a training bot is a choice about model training: AI search has its own bots, set on their own lines.
| robots.txt name | Run by | Used for | Kind |
|---|---|---|---|
| Googlebot | AI Overviews, AI Mode | AI search | |
| Bingbot | Microsoft | Bing search, Copilot | AI search |
| OAI-SearchBot | OpenAI | ChatGPT search | AI search |
| Claude-SearchBot | Anthropic | Claude search | AI search |
| PerplexityBot | Perplexity | Perplexity answers | AI search |
| ChatGPT-User | OpenAI | ChatGPT page visitsA request, not a rule: its maker says robots.txt may not apply when a user asks for the page. | Page visit a user asked for |
| Claude-User | Anthropic | Claude page visits | Page visit a user asked for |
| Perplexity-User | Perplexity | Perplexity page visitsA request, not a rule: its maker says robots.txt may not apply when a user asks for the page. | Page visit a user asked for |
| GPTBot | OpenAI | OpenAI model training | Model training |
| ClaudeBot | Anthropic | Anthropic model training | Model training |
| Google-Extended | Gemini training and grounding, not SearchA name in robots.txt, not a crawler of its own. | Model training | |
| Applebot-Extended | Apple | Apple Intelligence trainingA name in robots.txt, not a crawler of its own. | Model training |
| CCBot | Common Crawl | An open dataset many models train on | Model training |
| meta-externalagent | Meta | Meta AI training | Model training |
Know first when an AI bot gets locked out
A new robots.txt rule or a plugin update can shut out ChatGPT search without anyone noticing. PerfBee compares each report with the one before. When one of the 8 AI search and answer bots could read your homepage last time and cannot now, you hear about it: in the app on every plan, and by email on Starter and up. Training bots are left out, because blocking them is a choice.
- Googlebot
- Bingbot
- OAI-SearchBot
- Claude-SearchBot
- PerplexityBot
- ChatGPT-User
- Claude-User
- Perplexity-User
What this check does not do
- It does not count how often AI tools mention or cite your brand, and it sends no prompts to them.
- It does not test firewall or CDN blocks. PerfBee never sends another company’s crawler user agent, so a block set in Cloudflare or another CDN does not show in robots.txt. The report says where to check it.
When it runs and what it costs
The check runs after every completed detailed crawl, on every plan, Free included. It uses no credits of its own: you pay only for the pages of the detailed crawl. Schedule the crawl and the check repeats by itself. Claude and ChatGPT can read the report too, through the PerfBee MCP server.
Questions about GEO and AI search readiness
Is this a GEO audit?
It is the technical part of one. Generative engine optimization (GEO) aims to get your pages quoted in AI answers. That starts with AI search tools being able to reach, read and quote them: robots.txt access, indexability, snippet rules, server-rendered text and structured data. PerfBee checks those after each completed detailed crawl: robots.txt once for the site, the rest page by page.
Is this the same as AI visibility tracking?
No. AI visibility trackers ask AI tools questions and count how often your brand appears. PerfBee checks whether those tools can read your site at all, which comes first.
Does blocking GPTBot remove my site from ChatGPT search?
Not according to OpenAI. GPTBot collects training data, while OAI-SearchBot is the crawler behind ChatGPT search. Each one is allowed or blocked on its own line in robots.txt.
Does Google-Extended affect Google Search?
No. Google says Google-Extended does not affect inclusion or ranking in Google Search. It decides whether Gemini may use your pages for training and grounding.
Can I ask Claude or ChatGPT for the report?
Yes. Connect PerfBee to Claude or ChatGPT and ask, for example, “Which AI bots can read my site?” The assistant reads your latest report through the get_ai_readiness tool.
See what AI crawlers see.
Run a detailed crawl of your site. The AI readiness report follows it.
On every plan. The Free plan needs no card.