Skip to content
AI search

AI Readiness: How to Read the Report

5 min read · Updated Nov 22, 2026

Whether AI search engines can access and read your pages: robots.txt rules for 14 AI agents, pages an AI answer cannot quote, text that appears only after JavaScript, structured data and llms.txt. Every plan.

AI Readiness shows whether AI search engines can access and read your pages: ChatGPT, Claude, Perplexity and the AI features of Google search. It checks your robots.txt for 14 AI agents, lists the pages an AI answer cannot quote, and finds text that appears only after JavaScript. It also checks your structured data and your llms.txt. Every detailed scan writes the report. It is on every plan and costs no credits of its own.

Where to find it

Open AI Readiness in the site menu. It shows the report of the site's newest detailed scan. If the screen has two tabs, the report is the first one, AI Readiness. You can also open any detailed scan in Site Crawler and go to its AI Readiness tab.

A fast scan has no report, because it does not read page content. The report is ready about a minute after the crawl finishes.

The four numbers at the top

  • AI answer bots allowed: of the 8 AI agents that answer questions, how many your robots.txt lets read your homepage.
  • Pages AI can quote: the indexable pages that nothing blocks.
  • Render gap: sampled pages that need JavaScript to show their key text.
  • Brand checks: how many of the two brand checks pass (see below).

The 14 AI agents and your robots.txt

The table AI bots and robots.txt has one row for each agent, with its Operator, its Purpose, what your robots.txt says and what it is Used for. PerfBee checks the rules for each agent against every page the crawl found. A row says Allowed, Blocked, or how many of your pages are blocked. A group in robots.txt that names an agent applies to it instead of the * group.

  • Search (5 agents): Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot.
  • User fetch (3 agents): ChatGPT-User, Claude-User and Perplexity-User. They fetch a page a user asked for.
  • Training (6 agents): GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent.

The 8 search and user fetch agents are the ones that answer questions. Blocking a training agent is a fair choice, so the top numbers and the alert below count only those 8. Two notes the table shows too:

  • OpenAI and Perplexity say their user agents (ChatGPT-User and Perplexity-User) may not follow robots.txt when a user asked for the page. The table marks them as a request, not binding.
  • Google-Extended and Applebot-Extended are names in robots.txt, not crawlers. Googlebot and Applebot fetch the pages. These names say whether Gemini and Apple may use them.

Beside the table, Content-Signal says whether your robots.txt states how your content may be used. It is optional.

Pages AI answers cannot quote

Page issues by folder lists every problem by kind. Pick a kind, then open a folder to see its pages. Four problems stop an AI answer from quoting a page, and they count against Pages AI can quote:

  • Not indexable: the page says noindex.
  • Snippets disabled: the page says nosnippet.
  • max-snippet:0: the page allows no text in a snippet.
  • Content needs JavaScript: the page has fewer than 100 words, and its app container is almost empty until a script fills it. Pages the crawl rendered with JavaScript are not judged this way.

The warnings do not block a quote, but they limit what an AI answer can take: Snippet length limited, Duplicate structured data, Invalid JSON-LD and Incomplete structured data.

Text that appears only after JavaScript

Many AI agents do not run JavaScript. Render gap opens two pages in a browser: your homepage and a page from another of your top folders. For each, it compares the words in the raw HTML with the words after JavaScript runs, and shows the share that is hidden. A page gets a warning from 20% hidden, and is marked as a problem from 50%. Render that text on the server so AI answers can quote it.

Structured data

  • Readable without JavaScript: for pages with Product markup, whether the structured data states a price, a currency and an availability. An AI agent that does not run JavaScript needs them in the HTML.
  • Brand identity: Organization schema on the homepage, as JSON-LD, and Social profiles (sameAs) that tie your brand name to your domain. These are the two brand checks.
  • Duplicate, invalid and incomplete structured data on any page show in Page issues by folder, as above.

llms.txt

Brand identity also says whether your site has an llms.txt file: a short guide to your site for AI tools. PerfBee checks that the file answers and is not an HTML page. It does not read what the file says. The file is optional, Google ignores it, and it is not one of the two brand checks.

When an AI bot loses access

After each detailed scan, PerfBee compares your robots.txt with the previous detailed scan of the same address. If one of the 8 agents that answer questions could read your homepage before and cannot now, you get the alert AI crawlers lost access. It names the agents that lost access. Training agents never set it off.

The alert comes in the notification bell, by email, and as a webhook if you add one. Email alerts are on every plan. You can turn it off in Settings, Notifications.

Download, API and AI apps

  • Download: the page issues as CSV or Excel, one issue or all of them, with the page, the folder, the detail and how to fix it. Downloads are on every plan.
  • API: GET /crawls/{crawl_id}/ai-readiness returns the report of a detailed crawl.
  • Claude and ChatGPT: with the PerfBee MCP server, the get_ai_readiness tool reads the same report. Ask "Can AI search engines read my site?".

AI Readiness reads what your robots.txt and your pages say. To see which AI crawlers actually came to your site, use Bot Tracker (Starter and up).