ngl ran into this exact issue last quarter when migrating a massive ecom site to a decoupled architecture. the crawler was essentially treating each subdomain as a separate entity and completely failing to respect the global directives. we found that even with the proper paths defined, the
secondary passes were only hitting the root sitemap index and skipping the nested files entirely.
>it's like the bot just gives up halfway through the crawl budget.we had to consolidate everything into a single, massive sitemap index at the root level to force the handshake. have you tried verifying if the
sitemap:
directive in the subdomain-specific robots. txt is actually being read during those heavy rendering windows? might be worth checking the logs for 404s on the individual
.xml
files specifically.