Blocking spoofed AI user agents did not clean my analytics — a month of numbers

On 8/3 I put two Cloudflare WAF rules in front of this site. One blocks credential-scanning paths like /.env, the other blocks five user agents that had fetched zero articles and only scanned. The scanners were claiming AI crawler names, which meant my AI traffic numbers were partly a record of someone else’s port sweep. Stopping them at the edge would fix the damage and the measurement at the same time.

The second half of that turned out to be wrong. Over the following month, scans claiming AI user agents went from 145 to 991 in an eight-day window. Perplexity-User, which the rules were blocking the whole time, still showed up as 595 requests. And the scanning had spread onto Amazonbot and ChatGPT-User — names that real crawlers also use.

A blocked request is still a counted request

Cloudflare’s httpRequestsAdaptiveGroups aggregates what the edge responded, so a rule turns a request into a 403 rather than removing it from the data.

Perplexity-User shows this cleanly. Before the rule it arrived and got 404s. After the rule it got 403s, and the volume went up.

The rule works. The 404s are gone, so nothing reaches the origin. But as long as “AI crawler traffic” is counted by user agent, those 595 requests sit inside the number. A WAF protects the origin; it does not correct a metric. Those are two jobs and I had merged them.

The setup and the two rules

A static Astro site on Cloudflare Pages, free plan zone. The rules go in through the API from a script that is idempotent — it matches on description, updating the rule if it exists and creating it if not.

The first rule matches on path and ignores the user agent.

SCAN_PATH_TOKENS = [
    "/.env", "/.git", "/.aws", "/.svn", "/.ssh",
    "/wp-", "wp-admin", "wp-login", "wp-includes", "wp-config",
    "wlwmanifest", "xmlrpc.php",
    "secrets.json", "service_account", "/actuator/", "/api/auth/session",
    "phpinfo", "/.npmrc", "/.htpasswd", "id_rsa", "config.json",
]

scan_expr = " or ".join(
    f'(http.request.uri.path contains "{t}")' for t in SCAN_PATH_TOKENS
)

contains matters here. The zone already had a hand-written rule using starts_with(http.request.uri.path, "/.env"), which only matches at the root. The requests actually arriving were /server/.env (18 of them), /admin/.env and /app/.git/HEAD, so a rule existed and caught none of them.

The second rule matches on user agent, gated on Cloudflare not having verified the client.

SPOOFED_ONLY_UAS = [
    "Bytespider", "cohere-ai", "anthropic-ai", "Perplexity-User", "Google-Extended",
]

ua_expr = "(not cf.client.bot) and (" + " or ".join(
    f'http.user_agent contains "{u}"' for u in SPOOFED_ONLY_UAS
) + ")"

Those five were the ones with zero article fetches and a 0% verification rate at the time. CCBot and PerplexityBot stayed off the list: both verified at 0% too, but both were fetching real pages with 200s, and cutting off whoever collects training data only reduces the site’s reach.

The blocked name was kept, and real names were added

The thing that changed over the month was not the rules but what the scanner called itself. Grouping scan-classified requests by claimed user agent:

The blocked name stayed in use, and unblocked names were layered on top of it: Amazonbot from 67 to 284, ChatGPT-User from 51 to 181. The paths did not diversify — the same scanning simply arrived under more names.

Those two names are the problem, because real crawlers send them. In the same eight days Amazonbot split 135 verified against 489 unverified, and ChatGPT-User split 211 against 336. Matching the UA string would drop the genuine ChatGPT-User that fetched 216 articles along with the fake ones. A blocklist keyed on user agent expires the moment the impersonation moves to a name you need.

Names that cannot exist are a different case. Google’s crawler documentation states that Google-Extended has no HTTP request user agent — it is a robots.txt control token — so every request claiming it is fake and blocking costs nothing. That is the only shape where a UA rule holds up.

A path blocklist is maintained after the fact

Rule one ignores the user agent, so it stops a match no matter who claims to be sending it. It stops exactly the tokens I wrote down, though.

The 08-28 snapshot still had 300 requests from AI-claiming UAs ending in 404. They were hitting /secrets.yml, /.zshrc, /id_ecdsa and /@fs/proc/self/environ, none of which are in the token list. I wrote secrets.json but not .yml; I wrote id_rsa but not id_ecdsa.

The list grows by reading what arrived. config.json started as an exact eq "/config.json" match and was walked around with /runtime-config.json and /api/runtime-config.json. It only became a contains after I checked that none of the 164 real paths in the sitemap contain that string.

Every token you add also widens the chance of catching something you actually serve. Checking new tokens against the sitemap before adding them keeps that from turning into an outage.

The drop was not the rules

In the 09-05 snapshot, scans claiming AI user agents fell to 27. Total requests from those UAs went from 2,024 to 688, and ChatGPT-User, Amazonbot, GPTBot and OAI-SearchBot all came back 100% verified. Read on its own, that looks like the mitigation finally landing.

I changed nothing in those eight days, and these rules do not have a delayed effect.

Meanwhile zone-wide 403s went from 2,754 to 4,137. The scanning did not stop; it stopped using AI names. I cannot prove that from one snapshot, but writing “the rules brought it down” would be attributing someone else’s decision to my own change.

When judging a mitigation, put the date you changed something next to the date the number moved. If they do not line up, the cause is outside your change. Here they did not line up.

What I run now

Blocking and measurement live in separate layers.

flowchart TD
  Req["Request claiming an AI user agent"] --> WAF{"Matches a WAF rule?"}
  WAF -->|"match"| B["403 at the edge, origin untouched"]
  WAF -->|"no match"| O["Origin responds 200 / 404"]
  B --> A["Both are counted in analytics"]
  O --> A
  A --> C{"Classify at collection time"}
  C --> Y["Verified × real path = the number to read"]
  C --> N["Unverified / scan path = discard"]

The WAF is there to keep traffic off the origin and 404s out of the logs. Cleaning the numbers is not its job.

Collection handles that instead, splitting on two axes: whether verifiedBotCategory is non-empty, and whether the path exists in the sitemap. The previous post has that classifier in full. The scan series in this post is just its scan column lined up across snapshots.

Keeping the layers apart means editing a WAF rule does not move the measurement series, and changing the classifier does not change what gets blocked. After a month, that independence is the part I would keep.

Wrapping up

The WAF earns its place: nothing reaches the origin and the 404 pile shrank. The half I got wrong was expecting the analytics to get cleaner. Blocked requests stay in the data as 403s, so contamination has to come out at collection time.

The other lesson is that user-agent blocking has a short shelf life. Names that cannot exist — Google-Extended, anthropic-ai — stay blockable indefinitely. Once the impersonation moves to a name real crawlers share, matching the string is no longer available to you, and judging by the verification result instead is about the only way out.

For the other side of this — making the site readable to the crawlers I do want — there is the multilingual llms.txt post.

FAQ

Do WAF-blocked requests disappear from Cloudflare analytics?

No. The GraphQL Analytics API (httpRequestsAdaptiveGroups) aggregates the response the edge returned, so a blocked request stays in the data as a 403. In my logs, a user agent I was actively blocking accounted for 595 requests over eight days, every one of them a 403. Blocking keeps traffic off your origin; it does not remove it from your counts.

Is it safe to block a spoofed AI crawler by matching the User-Agent string?

Only for names that cannot exist. Google-Extended has no HTTP user agent of its own, so every request claiming it is fake and a UA match costs nothing. ChatGPT-User and Amazonbot are different: real crawlers send those exact strings, so a UA match would also drop genuine article fetches — and those are precisely the names the scanning moved to once I blocked the others.

How far does a path blocklist get you?

As far as the tokens you wrote. Common ones like /.env and /wp- stop, but paths outside the list — /secrets.yml, /.zshrc, /id_ecdsa, /@fs/proc/self/environ — pass through and become 404s; 300 of them remained in one eight-day window. A path blocklist is maintained after the fact, by reading what actually arrived.

If scans drop after I add a rule, did the rule work?

Not necessarily. Scans claiming AI user agents fell from 699 to 27 in a window where I changed no rules at all, while zone-wide 403s rose from 2,754 to 4,137. The scanning did not stop — it stopped using AI names. Line up when you changed something against when the number moved, and look outside your own changes when the two do not match.