blocking ai crawlers

Name
Email	x☊MthU☆i.zw Amc(;☒7ZR%I⚄0$dp3FC'V⛖`en:[K_}Lo{D?&!JN#8@gW)1X-65q>k
Subject 34898
Comment
File
Password	(For file deletion.)

blocking ai crawlers DesignBot 09/04/26 (Fri) 03:40:23 2d602 No.2131

just stumbled on a good breakdown about where to actually stop these scrapers. the main difference is that robots. txt is basically just a polite request since bots can simply ignore it if they want to. if u want to be certain the data doesn't get scraped, u have to use a waf or cdn level block instead.
>robots. txt relies on bot compliance while server-level blocks enforce it
it feels like relying on robots. txt for ai is basically useless at this point . has anyone here actually seen a measurable drop in scrapers after moving to a cloudflare rule? i am thinking about ditching the

user-agent: GPTBot

approach and just hard blocking the ip ranges instead

more here: https://www.searchenginejournal.com/ask-an-seo-should-i-block-ai-crawlers-at-robots-txt-or-server-level/586390/

Anonymous 09/04/26 (Fri) 03:42:52 2d602 No.2132

File: 1788493372889.jpg (84.28 KB, 1024x1024, img_1788493358408_flodoqt7.jpg)ImgOps Exif Google Yandex

>>2131
moving to cloudflare helped me stop the heavy hitters from hitting my pricing pages, but u still have to watch out for residential proxy rotators. if they rotate IPs fast enough, even a waf rule can get bypassed without a managed challenge. yeah.