AI crawler user agents are self-reported: 468 real fetches, 991 fake ones

Every eight days I pull the AI crawler numbers for this site: how many requests from ChatGPT-User, how many from GPTBot, one row per User-Agent. Rising numbers meant the AI systems were picking the site up.

That reading is gone. In this week’s run, requests that actually fetched an article came to 468, while credential scans came to 991 — more than double. Of the requests calling themselves GPTBot, Cloudflare could verify only 13% as OpenAI. The series I had been watching was tracking attacker volume, not AI interest.

Check the claim against a verification result

A User-Agent is a header. Writing GPTBot/1.2 into it costs nothing, so a table grouped by UA is a table of claims.

Behind Cloudflare you get two things to check those claims against. The first is verifiedBotCategory, which carries Cloudflare’s own reverse-DNS verification result and is an empty string when the request could not be verified. The second is whether the requested path exists in your sitemap. Split on both and what remains is “verified bots fetching real pages” — the only figure worth putting in a report.

The setup

A static Astro site on Cloudflare Pages, free plan zone, queried from Python through the GraphQL Analytics API.

Piece	Purpose	Free plan
httpRequestsAdaptiveGroups	request aggregation	yes (max one day per request)
verifiedBotCategory	bot verification result	yes
clientAsn / botClass	source ASN, bot classification	not permitted
requests + pandas	fetch and aggregate	—

Retention on the free plan is short — daily granularity reaches back about a week. You cannot re-derive the past later, so the script writes a snapshot to disk on every run.

One UA string, two rows

Adding verifiedBotCategory to the dimensions is the whole trick. Eight days of my logs:

The same string lands in two rows. Of 547 requests presenting as ChatGPT-User, 211 came from an address Cloudflare traced back to OpenAI.

Two agents came through clean: meta-externalagent and Applebot, both 100% verified with no impersonation mixed in. Those are the only two rows where the claimed number is usable as-is. At the other end, 455 Perplexity-branded requests were all unverified — not a single confirmed one in the window.

The query:

QUERY_BY_UA_VERIFIED = """
query ($zoneTag: String!, $since: Time!, $until: Time!) {
  viewer {
    zones(filter: { zoneTag: $zoneTag }) {
      httpRequestsAdaptiveGroups(
        limit: 5000
        filter: { datetime_geq: $since, datetime_leq: $until }
        orderBy: [count_DESC]
      ) {
        count
        dimensions { userAgent verifiedBotCategory }
      }
    }
  }
}
"""

A free zone returns at most one day per request, so the range gets chunked by the caller. Days past the retention window error out, so a failed day is skipped rather than fatal.

def fetch_rows(token, query, zone_tag, since, until):
    # split the range into one-day windows and concatenate the rows
    all_rows, cursor = [], since
    one_day = dt.timedelta(days=1)
    while cursor < until:
        win_end = min(cursor + one_day, until)
        variables = {
            "zoneTag": zone_tag,
            "since": cursor.isoformat() + "Z",
            "until": win_end.isoformat() + "Z",
        }
        try:
            all_rows.extend(rows_from(gql(token, query, variables, exit_on_error=False)))
        except RuntimeError as e:
            print(f"    [skipped] {cursor.date()}-{win_end.date()}: {e}")
        cursor = win_end
    return all_rows

One of the agents in my logs cannot exist

123 requests claimed Google-Extended. Verified share 0%, and 70 of them hit paths like .env or wp-login.

No inference is needed on this one. Google’s crawler documentation states that Google-Extended has no separate HTTP request user agent string: crawling happens under the existing Google user agents, and the token exists only to be addressed in robots.txt when you want to control AI training use.

So every request presenting that UA is, by definition, not Google. The same logic extends to any bot whose operator publishes IP ranges — OpenAI ships gptbot.json, and a reverse lookup settles the question.

On my zone all 123 were already stopped with a 403.

Classify the path, not just the client

Verification alone is not enough. A verified bot fetching robots.txt has read nothing. Five buckets:

SCAN_PATTERNS = (
    "wp-", ".env", ".git", ".aws", ".svn", ".ssh", "secrets", "credentials",
    "config.json", "service_account", "actuator", "api/auth", "phpinfo",
    ".bak", ".yml", ".yaml", ".php", ".sql", "id_rsa", ".npmrc", ".htpasswd",
)
OPS_PREFIXES = ("/robots.txt", "/sitemap", "/llms.txt", "/favicon", "/rss", "/feed", "/.well-known/")
ASSET_PREFIXES = ("/_astro/", "/images/", "/assets/", "/fonts/", "/cdn-cgi/", "/_image")


def classify_path(path, sitemap_paths):
    # content = a real page was consumed / ops = crawl bookkeeping
    # asset = static file / scan = credential probing / other = path does not exist
    if not path:
        return "other"
    low = path.lower()
    if any(k in low for k in SCAN_PATTERNS):
        return "scan"
    if low.startswith(OPS_PREFIXES):
        return "ops"
    if low.startswith(ASSET_PREFIXES):
        return "asset"
    if not sitemap_paths:
        return "unknown"   # cannot assert existence, so cannot call it content
    return "content" if (path.rstrip("/") or "/") in sitemap_paths else "other"

Using the sitemap as the source of truth for existence is what makes the classification hold up. Deciding from the response status is the obvious alternative, but redirects and paths that answer 200 without being real pages both leak into it. The set of paths you have declared public is a cleaner definition of “a page of mine”.

The unknown branch earns its place too. Fold it into content and the number spikes on any day the sitemap fetch fails, with nothing in the output to explain why.

flowchart TD
  Req["Request claiming an AI bot UA"] --> V{"verifiedBotCategory empty?"}
  V -->|"empty = unverified"| Fake["impersonation, drop from the count"]
  V -->|"verified"| P{"path in sitemap?"}
  P -->|"exists"| C["content = the number to report"]
  P -->|"robots.txt / sitemap"| O["ops = crawl bookkeeping"]
  P -->|".env / wp-login"| S["scan = credential probing"]

The result: 468 article fetches against 991 scans

Eight days, both axes applied. The content column is the one worth reading.

content totals 468 and scan totals 991. By status code, 403s came to 957 against 705 requests answered with 200 — the WAF is now blocking more than the site serves to these clients.

Six days earlier the same site read 610 content against 357 scan, so the ratio inverted inside a week. Grouped by UA alone, the story would have been “AI crawler traffic is up”.

The ClaudeBot row is worth pulling apart as well. Of 133 verified requests, 6 fetched articles and 129 fetched robots.txt and similar. “ClaudeBot sent 259 requests” and “six articles were read” are the same data.

What the split still cannot tell me

Both axes remove impersonation. Neither resolves this: content fell from 610 to 468, and I cannot say whether AI-side interest dropped or whether impersonation that used to land in content got reclassified as scan. GPTBot went from 85 content fetches to 7 while its verified share fell from 77% to 13%, so “fetched less” and “buried under fakes” are happening at once.

The aggregate output therefore keeps the verified share as a column, and any series whose share collapses gets marked unreadable for that snapshot. Two numbers side by side — the denominator and the verified share — beat one clean-looking figure.

Wrapping up

If you are reporting AI crawler numbers, distrust the UA first. One extra dimension splits claim from reality, and the shape of the split also tells you how far to trust that series this week.

What surprised me is which agents came through clean: meta-externalagent and Applebot, and nothing else. The better-known the bot, the more it gets impersonated, and the less usable its row becomes. Reputation of the operator turned out to say nothing about the quality of the data.

The same measure-first habit shows up in the post about measuring cross-post overlap. The other side of this work — making the site legible to AI clients in the first place — is in generating a bilingual llms.txt.

FAQ

Why can’t I count AI crawler traffic by User-Agent?

The User-Agent is a request header, so anyone can send GPTBot or ChatGPT-User in it. In eight days of my own logs, only 211 of the 547 requests calling themselves ChatGPT-User came from an IP Cloudflare could verify as OpenAI. Amazonbot was 135 verified out of 624. Count by UA alone and those unverified requests become your evidence that AI interest is growing.

Can I separate impersonators on Cloudflare’s free plan?

Yes. In the GraphQL Analytics API, add userAgent and verifiedBotCategory to the dimensions of httpRequestsAdaptiveGroups and you get Cloudflare’s reverse-DNS verification result per row, with an empty string meaning unverified. clientAsn and botClass need a paid plan, but verifiedBotCategory works on a free zone. The one constraint is that a single request covers at most one day, so loop over the range.

What should I do with requests claiming to be Google-Extended?

Treat every one of them as fake. Google’s crawler documentation states that Google-Extended has no separate HTTP request user agent string — it is a robots.txt control token, and crawling happens under the regular Google user agents. My logs had 123 requests claiming it, 0% verified, with 70 of them hitting credential-scanning paths.

Is a verified request the same as someone reading an article?

No, and separating the two matters. A verified bot fetching robots.txt, sitemap.xml or llms.txt is doing crawl bookkeeping, not reading. Of ClaudeBot’s 133 verified requests, 6 fetched articles and 129 fetched files like robots.txt. Classifying by whether the path exists in your sitemap splits content from ops and leaves only the number worth reading.