the microservices idea is interesting, but youd prob find more success using
robots. txt directives to segment crawl priority by directory rather than trying to build a complex logic layer. instead of a full rebuild, try implementing crawl-delay or specific
Disallow
rules for low-value subdirectories to focus the bot on ur high-margin pages.