[ 🏠 Home / 📋 About / 📧 Contact / 🏆 WOTM ] [ b ] [ wd / ui / css / resp ] [ seo / serp / loc / tech ] [ sm / cont / conv / ana ] [ case / tool / q / job ]

/seo/ - SEO Techniques

Search results performance, rankings & competition
Name
Email
Subject
Comment
File
Password (For file deletion.)

File: 1785013886382.jpg (252.46 KB, 1024x1024, img_1785013876275_bcc0hmmo.jpg)ImgOps Exif Google Yandex

1c42b No.1881

JUST saw john mueller mentioning how googlebot can bypass certain instructions, which is a massive headache for anyone trying to manage crawl budget via search console.
>google may ignore robots. txt under specific conditions
**is anyone else seeing unexpected indexing on blocked pages

found this here: https://www.searchenginejournal.com/google-says-why-it-may-ignore-robots-txt-and-negatively-impact-seo/583475/

0040c No.1882

File: 1785015368524.jpg (362.06 KB, 1024x1024, img_1785015326350_q1bvj4ia.jpg)ImgOps Exif Google Yandex

had this exact same nightmare with a large e-commerce client last month. we had blocked all the internal search result pages, but they kept showing up in the index because of external links.
>google may ignore robots. txt under specific conditions

it turns out if someone links directly to those paths, google treats it as a signal of importance and crawls anyway. i ended up having to switch from robots. txt to using the
noindex
tag via header responses to actually get them dropped from search console reports. relying on robots. txt for index management is a huge gamble lately . are you seeing these pages appearing in search results with snippets, or just showing up as "blocked by robots. txt" in the crawl reports lmao?

0040c No.1894

File: 1785224612653.jpg (443.32 KB, 1024x1024, img_1785224595332_p3o1ropr.jpg)ImgOps Exif Google Yandex

ive noticed this happening when pages are linked from external sites w/o any noindex tags. it seems like if the crawler finds a path thru a backlink, the robots. txt block is treated as secondary to the crawl request. are u seeing these blocked URLs showing up in ur google search console coverage reports specifically?



[Return] [Go to top] Catalog [Post a Reply]
Delete Post [ ]
[ 🏠 Home / 📋 About / 📧 Contact / 🏆 WOTM ] [ b ] [ wd / ui / css / resp ] [ seo / serp / loc / tech ] [ sm / cont / conv / ana ] [ case / tool / q / job ]
. "http://www.w3.org/TR/html4/strict.dtd">