
User-agent: * doesn't apply to crawlers you named — Googlebot fetched my disallowed links 133 times
I put Disallow: /go/ under User-agent: * and Googlebot and ClaudeBot kept fetching those links anyway. If a crawler has its own group, it never reads the * group. Cloudflare logs from before and after the fix, and why Python's urllib.robotparser can't check it.










