Skip to content

Bot Tracker

See what GPTBot, ClaudeBot and Googlebot actually did on your site.

Upload a server log or connect Cloudflare. Bot Tracker counts ten search and AI crawlers per day, crawler and page, and checks each logged visit against the addresses the crawler’s operator publishes.

Try it free for 14 days

On Starter and up. The Business trial includes it, with no card.

Real crawlers, proved.

Anyone can send a request that says it is Googlebot. Bot Tracker compares the address of each logged visit with the list the crawler’s operator publishes. A visit that matches is proved. One that does not is fake: it is shown on its own and left out of every other count.

The ten search and AI crawlers, per day, crawler and page.

  • Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User. One row for each, also the ones that never came.
  • For each crawler: visits, proved visits, visits answered with a page, and the last day it came.
  • For each AI crawler, what your robots.txt allows it, from your latest full scan, next to what the log shows: allowed and never came, came and was served, or turned away with a 401, 403 or 429.
  • Every screen shows the 30 days that end at the newest day in your logs, so a log you upload once a month never shows an empty month.
Pages and the log

Where your pages and the log disagree.

Bot Tracker sets the log next to your latest scan and lists the pages where the two do not match.

  • Never visited: pages a search engine may index that neither Googlebot nor Bingbot asked for in the days of the log.
  • Wasted visits: the crawl budget search bots spend on parameter URLs, on pages that say noindex or name another canonical, and on answers that were not a page.
  • Found by bots only: pages a bot got that no scan found, such as old addresses or pages nothing links to.
  • Site Pages gets three columns (search bots, AI bots and the last bot visit), and Page Detail gets a Bot visits card.
Server log or Cloudflare

Upload a server log, or connect Cloudflare.

Upload an access log whenever you like, or let PerfBee read your Cloudflare analytics once a day. Each source says how far its crawler names can be trusted.

  • An nginx or Apache access log, a Cloudflare Logpush file of HTTP requests, or a CloudFront or IIS log in W3C extended format.
  • Up to 50 MB a file, gzip or plain text. A site takes 10 uploads a day, and up to 3 of yours can be read at once.
  • The upload card gives a one-line command that keeps only the crawler lines, so your visitors’ lines never leave your server.
  • Cloudflare: a read-only token, read once a day, up to 30 days back. Cloudflare gives no visitor address, so those visits are counted by user agent only and never shown as verified.

Members and admins can upload a log. Only the workspace owner can connect Cloudflare. Viewers can read everything.

What PerfBee keeps from a log, and what it never keeps.

Kept: counts

  • For each day, crawler and page: how many visits, and how each was answered (a page, a redirect, a 401, 403 or 429, a 404 or 410, another 4xx, or a 5xx).
  • The page as a path, as it appears in your log. For an address with query parameters, only the names of the parameters, never the values.
  • For each upload: when it arrived, who sent it, its format and size, the lines read and kept, the days it covers, and a fingerprint so the same file is never counted twice.
  • From Cloudflare: the token, encrypted and never shown again, and the daily counts by crawler, page and status group.

Never kept

  • IP addresses. The address of each visit is used only to check the crawler’s claim, and is not saved.
  • User agents, log lines, referrers, cookies, query values, or any request of a human visitor.
  • Your file. Each line is read in memory, and the file is deleted when processing ends, whether it worked or not. A leftover file is removed within two hours.

The counts stay for as many days as your plan’s history, at most 365, and are deleted each night after that. They go when you delete the site or your account. Disconnecting Cloudflare deletes every count read from it.

What Bot Tracker shows and does.

Every Bot Tracker feature and what it gives you
FeatureWhat you get
Screens
CrawlersOne row for each of the ten crawlers: visits, proved visits, visits answered with a page, and the last day each came.
AI crawlers: allowed, and did they comeWhat your robots.txt says about each AI crawler, from your latest full scan, next to what the log shows.
Never visitedPages a search engine may index that neither Googlebot nor Bingbot asked for in the days of the log.
Wasted visitsThe crawl budget search bots spend on parameter URLs, noindex pages, pages that name another canonical, redirects and error answers.
Found by bots onlyPages a bot got that no scan of your site found. If your last scan did not cover the whole site, the list says so.
Site Pages and Page DetailThree columns in Site Pages (search bots, AI bots and the last bot visit) and a Bot visits card on the Overview tab of Page Detail.
Sources and proof
Server log uploadnginx, Apache, Cloudflare Logpush, CloudFront or IIS logs, up to 50 MB a file. 10 uploads a day for each site, 3 read at once. A file replaces the days it covers, or adds a second server’s log to them.
Cloudflare connectionA read-only token, read once a day, up to 30 days back. Counted by user agent only and kept apart from the numbers of an uploaded log.
Fake crawler checkEach logged visit compared with the addresses its crawler’s operator publishes: proved, fake, or not proved when the list was too old to trust.
CDN warningWhen a log carries a CDN’s address, Bot Tracker says so and counts every visit that calls itself a crawler.
Plan and access
PlanStarter and up, with no credits. Included in the 14-day Business trial.
HistoryAs many days as your plan’s history keeps, at most 365.
Team rolesMembers and admins upload logs, viewers read everything, and the workspace owner connects Cloudflare.

Bot Tracker, answered.

Anything else? Talk to us.

How do I track GPTBot and ClaudeBot visits?

Upload your server’s access log, or connect Cloudflare. Bot Tracker counts the visits of GPTBot, ClaudeBot and 8 more search and AI crawlers per day, crawler and page. Before you upload, run the one-line command from the upload card on your server: it keeps only the crawler lines.

Which crawlers does Bot Tracker count?

These ten: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User. Each one has a row, also when it never came. A training crawler may come only once a month, so a quiet crawler is not always a problem.

How does it tell a real Googlebot from a fake one?

It compares the address of each logged visit with the list of addresses the crawler’s operator publishes. A visit that matches is proved. A visit that does not match is fake: it is shown on its own and left out of every other count. If an operator’s list was too old to trust when your log was read, those visits are counted and marked as not proved.

Which log formats can I upload?

An nginx or Apache access log, a Cloudflare Logpush file of HTTP requests (one JSON object per line), or a CloudFront or IIS log in W3C extended format. A file is up to 50 MB, gzip or plain text, and the log needs the user agent of each visit.

Does PerfBee keep the IP addresses in my log?

No. The address of each visit is used only to check the crawler’s claim, and is not saved. PerfBee keeps counts per day, crawler and page, and never user agents, log lines, referrers, cookies or query values. The file is deleted when it has been read. The counts stay for as many days as your plan’s history, at most 365.

What does the Cloudflare connection read?

Once a day, how many requests came from the ten crawlers, by page address, crawler and status group, up to 30 days back. The token needs two read-only permissions. PerfBee never asks Cloudflare for IP addresses, reads no cookies or query strings, and never changes your Cloudflare account. Because Cloudflare gives no address, these visits are counted by user agent only.

Which plan includes Bot Tracker, and what does it cost?

Bot Tracker is on Starter and up and uses no credits. The 14-day Business trial includes it.

Which days does it show?

The 30 days that end at the newest day in your logs, not at today, because a log is usually uploaded once in a while. Every screen says which days it shows.

What if my site is behind a CDN or proxy?

The log may then show the address of the CDN and not the visitor. Bot Tracker says so in a banner and counts everything that calls itself a crawler. A log from your own server with the visitor address, or the CDN’s own log, gives a proved answer.

How is Bot Tracker different from AI Readiness?

AI Readiness reads your robots.txt and your pages after each completed detailed crawl, and says whether AI search engines can access and read them. Bot Tracker reads your log and shows which crawlers actually came, next to what robots.txt allows each AI crawler.

Find out which AI crawlers read your site.

  • No credits used
  • 14 days of Business free
  • No card