been seeing too many devs accidentally leaving user profiles or session logs in the robots.txt. it's not even funny when you see a bunch of/12345/ settings pages indexed. i started using a specific pattern to catch all these dynamic paths before they hit the index. it doesn't cover everything but it stops the most obvious leaks. you don't want googlebot scraping every single private directory just because someone forgot a noindex tag on a template. use this if your url structure has those predictable numeric ids or deep nested folders.
Disallow: /user/[0-9]+/Disallow: /account/settings/.*
it's
perfect pretty decent for basic cleanup. just don't rely on it as your only security layer because anyone can still find the links if they try. does anyone else use regex in their robots file or are you all still doing manual disallows?