[ Home / About / Contact / WOTM ] [ b ] [ wd / ui / css / resp ] [ seo / serp / loc / tech ] [ sm / cont / conv / ana ] [ case / tool / q / job ]

/seo/ - SEO Techniques

Search results performance, rankings & competition
Name
Email
Subject
Comment
File
Password (For file deletion.)

File: 1794809027578.jpg (67.83 KB, 1000x750, img_1794809018091_mb3xn923.jpg)ImgOps Exif Google Yandex

2312a No.2418

been thinking about this since i saw that bit about agents browsing just like real users. it's getting messy because you can't just rely on robots.txt anymore when your cdn or waff is basically overriding everything without telling the seo team. i had a client last week where they thought they were blocking training scrapers but they were actually nuking the retrieval bots that feed search answers. ngl it's a nightmare to coordinate with dev and security teams just to decide which crawler gets through. if you block an agent doing a product comparison, you might as well just delete vanish from the results entirely. we really need a way to distinguish between someone training a model and someone actually fetching data for a search query. it feels like we're moving toward this whole deliberate policy approach where legal and engineering have to be in the loop too. i don't know how i'm supposed to manage that without some kind of centralized dashboard. good luck getting an engineer to care about your crawler list

full read: https://www.searchenginejournal.com/ai-bot-management-why-seos-need-more-than-block-lists/597148/

ede73 No.2423

>>2418
>if you block an agent doing a product comparison, you might as well just delete vanish from the results entirely
had this exact mess with a local plumber last month because his dev team thought they were being smart by blocking all headless browsers. turned out we lost basically all our visibility in those new ai search snapshots overnight. it's a total catastrophe for small sites that actually need that traffic.

ede73 No.2439

>>2423 That headless browser block is such a classic mistake. It's almost impossible to keep up when security teams treat every automated request as a threat instead of looking at the user agent string or the specific IP ranges. For your client, were they using a specific WAF rule that was catching everything by default?



[Return] [Go to top] Catalog [Post a Reply]
Delete Post [ ]
[ Home / About / Contact / WOTM ] [ b ] [ wd / ui / css / resp ] [ seo / serp / loc / tech ] [ sm / cont / conv / ana ] [ case / tool / q / job ]