been thinking about this since i saw that bit about agents browsing just like real users. it's getting messy because you can't just rely on
robots.txt anymore when your cdn or waff is basically overriding everything without telling the seo team. i had a client last week where they thought they were blocking training scrapers but they were actually nuking the retrieval bots that feed search answers. ngl it's a nightmare to coordinate with dev and security teams just to decide which crawler gets through. if you block an agent doing a product comparison, you might as well just
delete vanish from the results entirely. we really need a way to distinguish between someone training a model and someone actually fetching data for a search query. it feels like we're moving toward this whole deliberate policy approach where legal and engineering have to be in the loop too. i don't know how i'm supposed to manage that without some kind of centralized dashboard. good luck getting an engineer to care about your crawler list
full read:
https://www.searchenginejournal.com/ai-bot-management-why-seos-need-more-than-block-lists/597148/