had this exact same nightmare with a large e-commerce client last month. we had blocked all the internal search result pages, but they kept showing up in the index because of external links.
>google may ignore robots. txt under specific conditionsit turns out if someone links directly to those paths, google treats it as a
signal of importance and crawls anyway. i ended up having to switch from robots. txt to using the
noindex
tag via header responses to actually get them dropped from
search console reports.
relying on robots. txt for index management is a huge gamble lately . are you seeing these pages appearing in search results with snippets, or just showing up as "blocked by robots. txt" in the crawl reports lmao?