Bot Tracker shows which crawlers asked for which pages of your site: Googlebot, Bingbot, ChatGPT, Claude and Perplexity. It reads a server log that you upload. A scan can only show what your pages look like. Your server log shows who actually came, and what your server answered. Bot Tracker is on Starter and up and uses no credits.
Where to find it
Open Bot Tracker in the site menu. If your site menu has no Bot Tracker item, open AI Readiness: Bot Tracker is its second tab. Without the site menu, it is the Bot Tracker tab of Site Crawler. Pick your site in the Site menu.
What you see
- Crawlers. One row for each of the ten crawlers, also the ones that never came: how many visits, how many were proved, how many were answered with a page, and the last day each came.
- AI crawlers: allowed, and did they come. What your robots.txt says about each AI crawler, from your latest full scan, next to what the log shows. A crawler can be allowed and never come, come and be served, or come and be turned away with a 401, 403 or 429 answer.
- Never visited. Pages a search engine may index that neither Googlebot nor Bingbot asked for in the days of the log.
- Wasted visits. Search-bot visits that bought nothing: addresses with query parameters, pages that say noindex or name another canonical address, and answers that were not a page (redirects, 4xx and 5xx).
- Found by bots only. Pages a bot got that no scan of your site found: old addresses, or pages that nothing links to. If your last scan did not cover the whole site, the list says so.
- Site Pages and Page Detail. Once a site has a log, the Pages view gets three columns (search bots, AI bots and the last bot visit), and Page Detail gets a Bot visits card on its Overview tab.
Every screen says which days it shows. They are the 30 days that end at the newest day in your logs, not at today, because a log is usually uploaded once in a while. A training crawler may come only once a month, so a quiet crawler is not always a problem.
Get a log file
Use the access log of your web server (nginx or Apache), your Cloudflare Logpush (the HTTP requests dataset, one JSON object per line), or a CloudFront or IIS log (W3C extended). Ask your host if you do not know where it is. The log needs the user agent of each visit.
If your site is behind Cloudflare, you can connect Cloudflare in place of an upload. PerfBee then reads the visits once a day, counted by user agent only.
Keep only the crawler lines before you upload. Run this on your server, in the folder of your access logs:
zgrep -hiE 'googlebot|bingbot|gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|perplexity-user' access.log* | gzip > bots.log.gz
The lines of your human visitors then never leave your server. The same command is on the upload card, with a Copy button.
Upload it
- Open Bot Tracker as above, and pick your site.
- On the Upload a server log card choose what happens to days you uploaded before: Replace the days this file covers (the default) or Add to the days this file covers (the log of a second server for the same days).
- Press Choose a log file. A file is up to 50 MB, gzip or plain text. A site takes 10 uploads a day, and up to 3 of your uploads can be waiting or being read at once.
- Wait while PerfBee reads it. A big file takes a few minutes, and you can leave the page: the upload goes on. When it is done you see how many lines were read and how many crawler visits were kept.
Lines that name another host are left out. A log that does not name the host is counted whole, so upload the log of this site only. Days older than your plan's history are left out too, and the upload says how many. Members and admins can upload. Viewers can read everything but cannot upload or forget an upload.
Real crawlers and fake ones
Anyone can send a request that says it is Googlebot. PerfBee compares the address of each visit with the list of addresses that the crawler's operator publishes. A visit that matches is proved. A visit that does not match is fake: it is shown on its own and left out of every other count. If an operator's list was too old to trust when your log was read, those visits are not proved. They are counted, and marked.
If your site sits behind a CDN or proxy, the log may show the address of the CDN and not the visitor. PerfBee then cannot tell real crawlers from fake ones, says so in a banner, and counts everything that calls itself a crawler. A log from your own server with the visitor address, or the CDN's own log, gives a proved answer.
What PerfBee keeps, and what it does not
Your file waits on PerfBee's server only until it has been read. Each line is processed in memory and the file is deleted when processing ends, whether it worked or not. If something goes wrong, a leftover file is removed within two hours. What PerfBee keeps from a log is counts:
- For each day, each crawler and each page: how many visits, and how many were answered with a page (2xx, or 304 Not Modified, which means the crawler's copy is still good), a redirect (3xx), a 401, 403 or 429, a 404 or 410, another 4xx, or a 5xx.
- The page as a path: the part of its address after your domain. A path is stored as it appears in your log, so it can contain what your site puts in its URLs. For an address with query parameters PerfBee keeps only the names of the parameters, such as
sizeandcolor, never the values. Visits of fake crawlers and requests for images, scripts and other files are counted without a path. - For each upload: when it arrived, who uploaded it, its format and size, how many lines were read and kept, the days it covers, and how far the check of the crawlers could be trusted. PerfBee also keeps a fingerprint of the file, so the same file is not counted twice.
PerfBee does not keep IP addresses, user agents, log lines, referrers, cookies, query values or any request of a human visitor. The address of each visit is used only to check the claim of the crawler, and is not saved. The counts stay for as many days as your plan's history (at most 365 days) and are then deleted each night. They are deleted when you delete the site or your account. Large amounts are finished by PerfBee's nightly clean-up, at the latest the next night. Forget on an upload removes it from your list and deletes its file. The counts it added belong to the site, so they stay until they leave your plan's history. For the same time PerfBee keeps the fingerprint of the file and the time of the upload, so a forgotten file is not counted twice and still counts toward the 10 uploads a day.
Common questions
- The upload says it has no crawler visits. The file has no lines from the ten crawlers inside your plan's history. Check that it covers recent days, and that the user agent is in the log.
- The upload says the format is not recognised. Send an nginx or Apache combined log, a Cloudflare Logpush HTTP requests file, or a W3C extended log.
- The file is over 50 MB. Run the command above first. Crawler lines are a small part of a log.
- A page of mine shows a dash in the Bot visits columns. A page with a query string, or a file that is not a page, has no number of its own, because a log keeps only the names of parameters.
- The same file again. A file you already uploaded for a site is refused, so a log is never counted twice. Send a newer log.