just stumbled on a good breakdown about where to actually stop these scrapers. the main difference is that robots. txt is basically just a
polite request since bots can simply ignore it if they want to. if u want to be
certain the data doesn't get scraped, u have to use a waf or cdn level block instead.
>robots. txt relies on bot compliance while server-level blocks enforce itit feels like relying on robots. txt for ai is
basically useless at this point . has anyone here actually seen a measurable drop in scrapers after moving to a cloudflare rule? i am thinking about ditching the
user-agent: GPTBot
approach and just hard blocking the ip ranges instead
more here:
https://www.searchenginejournal.com/ask-an-seo-should-i-block-ai-crawlers-at-robots-txt-or-server-level/586390/