Measuring
Bot detection
Bot traffic is labelled and kept, never silently discarded. Human metrics always filter it out, and "which of my pages is GPTBot reading" stays answerable.
The distinction that matters
| AI crawler | AI referral | |
|---|---|---|
| What it is | GPTBot, ClaudeBot, PerplexityBot fetching your page | A real browser arriving from chatgpt.com |
| Is there a human? | No | Yes |
| Counted as a visitor? | Never | Yes |
| Where you see it | ai.crawler.* metrics | The ai channel |
Most tools conflate these. If they are shown together, "I am getting AI traffic" is both inflated and misread — you cannot tell whether a model read you or sent you someone.
Categories
| Kind | Examples |
|---|---|
ai-crawler | GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, CCBot, Meta-ExternalAgent, Applebot-Extended, Amazonbot, Diffbot |
search-crawler | Googlebot, bingbot, DuckDuckBot, YandexBot, Baiduspider, Applebot |
seo | AhrefsBot, SemrushBot, MJ12bot, DotBot, Screaming Frog, DataForSeoBot |
preview | facebookexternalhit, Twitterbot, Slackbot, Discordbot, LinkedInBot, WhatsApp |
monitor | UptimeRobot, Pingdom, StatusCake, Lighthouse, GTmetrix |
generic | HeadlessChrome, Puppeteer, Playwright, curl, python-requests, and an empty user agent |
An empty user agent counts as a bot. Real browsers always send one, so an empty value is either a script or a tool hiding itself — counting it as human would inflate your numbers silently.
Why bots are stored, not dropped
Umami and Plausible throw bot hits away. We keep them because the question "which pages are AI models actually reading" is the only real feedback loop for GEO/AEO work — you cannot optimise for citation if you cannot see what was fetched.
The safety is at the query layer, not at ingest: every human metric carries
bot_kind = '', and that constant is used rather than copy-pasted so it
cannot be forgotten in one place.
SELECT COUNT(DISTINCT visitor_id) AS value
FROM events
WHERE site_id = ? AND ts >= ? AND ts < ?
AND bot_kind = ''
A versioned table
Detection is a deterministic rule table with a version stamp, not a model. Each row records the table version that labelled it, so a reclassification is visible rather than retroactive. A golden-set eval runs in CI and has to stay at 100% — when a new assistant is added, the old rows are proven unbroken.
Quota
Crawler requests are stored but do not count against your plan's event quota. You did not ask GPTBot to visit, so you should not pay for it.