User-agent: * doesn't apply to crawlers you named — Googlebot fetched my disallowed links 133 times

I had Disallow: /go/ under User-agent: *, and Googlebot ignored it. /go/ is this site’s redirector to Amazon, and I don’t want search engines or AI crawlers following it. I added the rule on October 6 and assumed that was the end of it.

Three weeks later the Cloudflare logs showed verified Googlebot fetching those redirect links 133 times in three days, and verified ClaudeBot 213 times in five. The cause was higher up in the same file: I had separate User-agent: Googlebot and User-agent: ClaudeBot groups. A crawler that has a group of its own never reads the * group.

All times below are UTC.

Named groups replace the * group

A crawler picks the one group in robots.txt that matches its name. User-agent: * is only used when no group matches. It is the fallback for crawlers you didn’t name, not a base layer that applies to everyone.

RFC 9309, section 2.2.1, says that if no matching group exists, crawlers MUST obey the * group. When a matching group exists, * simply doesn’t come into it. Google’s robots.txt spec is more direct: only one group is valid for a particular crawler, and the rest are ignored.

Here is the old file, cut down:

User-agent: Googlebot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Allow: /
Disallow: /go/

I wanted the AI search crawlers explicitly allowed, so the file listed 13 named groups: Googlebot, bingbot, Applebot, GPTBot, ClaudeBot and so on, each with a single Allow: /. The /go/ line came later, and I put it under * because I thought that was the place for rules meant for every crawler.

flowchart TD
  Bot["Crawler reads robots.txt"] --> Q{"Is there a group with its name?"}
  Q -->|"Yes (Googlebot, ClaudeBot, ...)"| G["Uses only that group<br/>Allow: / so /go/ is allowed"]
  Q -->|"No"| W["Uses User-agent: *<br/>Disallow: /go/ so it is blocked"]

So the rule only stopped crawlers I had never named. The search engines and AI crawlers I actually cared about went straight through.

What the logs showed

I pulled every request to /go/ from Cloudflare’s GraphQL Analytics API (httpRequestsAdaptiveGroups), grouped by user agent, verifiedBotCategory and status code. verifiedBotCategory is Cloudflare’s own check against IP and reverse DNS, and it is empty for spoofed user agents (how I use it). Every Googlebot, ClaudeBot and bingbot request counted below was verified.

The Googlebot window is short because Cloudflare’s free plan keeps detailed logs for only eight days. I save a weekly snapshot for AI crawlers, but Googlebot wasn’t on that list. It was very likely doing the same before October 25. I just don’t have those numbers.

The redirect links already had rel="sponsored nofollow", and the responses carried X-Robots-Tag: noindex, nofollow. Neither stops the fetch. Since March 1, 2020 Google treats nofollow as a hint for crawling, so whether to follow a link is Google’s call. X-Robots-Tag arrives with the response, which means the request has already been made.

robots.txt is the only one of the three that stops the request itself, and it wasn’t working.

The fix: copy the rule into every group

Rules aren’t inherited between groups, so the only fix is to write the blocked paths into each named group. At 13:31 on October 27 I pushed a version that adds two lines to each of the 13 groups.

# Named groups below override "User-agent: *" entirely, so each repeats the
# Disallow lines (affiliate redirector /go/ and the decorative /_code-lines.json).

User-agent: Googlebot
Allow: /
Disallow: /go/
Disallow: /_code-lines.json

User-agent: ClaudeBot
Allow: /
Disallow: /go/
Disallow: /_code-lines.json

Having Allow: / and Disallow: /go/ in the same group is fine. RFC 9309, section 2.2.2, picks the rule with the longest match when several apply. For /go/amazon/..., both / (one character) and /go/ (four) match, and the longer Disallow: /go/ wins.

The comment at the top is a note for myself. Next time I need to block something, I will probably try the same one-line fix under * again.

After the fix: zero, once Googlebot re-read robots.txt

Googlebot’s last request to /go/ was in the 17:00 hour on October 27. From then until November 2 there were none.

Hour by hour, the fix didn’t land at once. After the push, Googlebot still fetched /go/ twice in the 15:00 hour and twice in the 17:00 hour. It had read robots.txt in the 13:00 hour, but that is the same hour as the push, so I can’t tell which version it got. The 15:00 requests suggest the old one. Its next robots.txt fetch was in the 17:00 hour, and that was the last time it touched /go/. Google caches robots.txt for up to 24 hours, so a delay like this is expected.

bingbot followed the same pattern: one more request at 20:00 after the push, then nothing.

ClaudeBot doesn’t fit the story. Its last /go/ request was at 18:00 on October 25, two days before the fix. It kept fetching articles after that, so it hadn’t left the site. I can’t credit the fix for this one; it may simply not have reached the redirect links during that week.

GoogleOther came too, without being named

One crawler that should have been covered by * also fetched /go/. GoogleOther is Google’s general-purpose crawler, and my robots.txt never names it. It still fetched the redirect links 7 times between October 6 and 9, and 9 times between October 20 and 24, all verified.

I don’t know why. If it used the Googlebot group that would explain it, but Google’s crawler documentation says nothing of the kind. It hasn’t come back since the fix, but it had already stopped on October 24, so that proves nothing. The counts are small, and I’m leaving this one open.

Python’s urllib.robotparser gets this wrong

When I tried to confirm the fix locally, I hit a second problem. Python’s standard urllib.robotparser reports every user agent as allowed, even on the fixed file.

from urllib.robotparser import RobotFileParser

AGENTS = ["Googlebot", "bingbot", "ClaudeBot", "GPTBot", "GoogleOther", "SomeNewBot"]
PATH = "/go/amazon/B0CL6H35DY"

rp = RobotFileParser()
rp.parse(open("robots_after.txt", encoding="utf-8").read().splitlines())
for ua in AGENTS:
    print(ua, "allow" if rp.can_fetch(ua, PATH) else "DISALLOW")

The standard library reads a group from the top and stops at the first line that matches. Allow: / comes first, so it never gets to Disallow: /go/. That isn’t the longest-match rule that RFC 9309 and Google use.

Protego, the parser Scrapy uses, does longest match. I used it (version 0.7.0) to compare the two files:

from protego import Protego

URL = "https://aulvem.com/go/amazon/B0CL6H35DY"

for f in ["robots_before.txt", "robots_after.txt"]:
    rp = Protego.parse(open(f, encoding="utf-8").read())
    print(f)
    for ua in AGENTS:
        print(" ", ua, "allow" if rp.can_fetch(URL, ua) else "DISALLOW")

The “Before” column matches the logs: crawlers with a group of their own got through, and unnamed ones were blocked. On paper GoogleOther is blocked, which is why I can’t explain its visits.

If your robots.txt has a lot of named groups, a table like this makes a forgotten group easy to spot: it is the row that says allow. It would be cheap to run in CI whenever robots.txt changes.

Wrapping up

A rule under User-agent: * reaches every crawler only if no crawler has a group of its own. Plenty of sites now allow AI crawlers by name, so any of them that later added a Disallow under * may be in the same state mine was.

There are two ways to check. One is to run each named user agent through a longest-match parser. The other is to look at your edge logs and see whether verified crawlers are still hitting the paths you meant to block. I noticed the problem through the logs, and used the parser to confirm the fix.

The earlier posts cover telling real AI crawlers from spoofed ones and what happened after I blocked the spoofed ones with a WAF.

FAQ

Does a Disallow under User-agent: * apply to Googlebot?

Not if the file also has a User-agent: Googlebot group. Under RFC 9309 a crawler follows the group that matches its own name and falls back to * only when no group matches. Google’s documentation says only one group is valid for a given crawler. If the named group only says Allow: /, the whole site is allowed for that crawler.

Do I need to repeat the same Disallow in every named group?

Yes. Rules are not inherited between groups, so every path you want blocked has to be written into each named group. I added Disallow: /go/ to each of the 13 named groups on this site. The more groups you have, the easier it is to miss one, so a check that runs every user agent through a parser is worth having.

Will Googlebot skip a link marked rel=“nofollow”?

Not necessarily. Google announced that from March 1, 2020 it treats nofollow as a hint for crawling and indexing too. The redirect links here carried rel="sponsored nofollow", and Googlebot still fetched 67 different redirect URLs, 133 requests in total. If you need crawling to stop, use robots.txt.

Can I test robots.txt with Python’s urllib.robotparser?

Not if a group mixes Allow and Disallow. The standard library parser takes the first matching line in a group, so with Allow: / written first it reported every user agent as allowed, even on the fixed file. RFC 9309 and Google use the longest matching rule instead. Protego, the parser Scrapy uses, follows that rule and showed the before and after correctly.