[ Home / About / Contact / WOTM ] [ b ] [ wd / ui / css / resp ] [ seo / serp / loc / tech ] [ sm / cont / conv / ana ] [ case / tool / q / job ]

/seo/ - SEO Techniques

Search results performance, rankings & competition
Name
Email
Subject
Comment
File
Password (For file deletion.)

File: 1794559821175.jpg (129.7 KB, 1206x566, img_1794559811882_ltnbzff3.jpg)ImgOps Exif Google Yandex

f3f9b No.2407

been seeing too many devs accidentally leaving user profiles or session logs in the robots.txt. it's not even funny when you see a bunch of/12345/ settings pages indexed. i started using a specific pattern to catch all these dynamic paths before they hit the index. it doesn't cover everything but it stops the most obvious leaks. you don't want googlebot scraping every single private directory just because someone forgot a noindex tag on a template. use this if your url structure has those predictable numeric ids or deep nested folders.

Disallow: /user/[0-9]+/Disallow: /account/settings/.*


it's perfect pretty decent for basic cleanup. just don't rely on it as your only security layer because anyone can still find the links if they try. does anyone else use regex in their robots file or are you all still doing manual disallows?

7ee05 No.2414

>>2407
>don't rely on it as your only security layer

That's the part people always ignore and then wonder why their data is in a search result. I once saw a client try to hide an entire
/admin/
directory with just a robots rule, which is brilliant basically an invitation for anyone to find the path.

7ee05 No.2417

>>2407
>Disallow:/0-9+/
regex doesn't even work in a standard robots.txt file. most bots just treat that as a literal string and you end up blocking nothing or everything. are you using some kind of custom middleware to rewrite these rules before they hit the crawler? im seeing people trip up on this all the time.

f7458 No.2432

>>2417
I've seen this exact mistake happen on a few staging sites I managed for clients. They thought they were being clever with some wildcard patterns but it just ended up blocking the entire product catalog instead of the private stuff. Most bots are way too literal to handle actual regex like that in a standard file.



[Return] [Go to top] Catalog [Post a Reply]
Delete Post [ ]
[ Home / About / Contact / WOTM ] [ b ] [ wd / ui / css / resp ] [ seo / serp / loc / tech ] [ sm / cont / conv / ana ] [ case / tool / q / job ]