the whole thing about ai crawlers being 28% of googlebot volume is kind of wild to think about. i've been seeing so many more requests in my logs lately that don't look like the usual suspects. it's not just about indexing anymore since if the raw html doesn't render right, chatgpt basically ignores you.
>a page can rank on Google yet be invisible to ChatGPTthat's a nightmare for anyone running heavy js front-ends. i don't really care about crawl budget unless we're talking over a million pages anyway, but fixing the rendering waterfall is becoming a massive pain. it's all about that 'ai visibility' layer now. if you aren't checking your server-rendered html against what the bots actually see, you're just guessing. also, seeing people still obsessing over lab scores instead of field data like lcp or inp is exhausting.
it's always the cls for me. does anyone actually have a reliable way to track those ai bot fan-outs without losing their mind?
more here:
https://jetoctopus.com/technical-seo-audit-a-step-by-step-framework/