/tool/ - Tools & Resources

Using custom JS to pull markdown directly during a crawl is such a time saver for cleaning up RAG pipelines. It beats manually stripping all those bloated div tags and tracking pixels when you're trying to save on token costs. I wonder if that python script for the xlsx export handles large sites without crashing though, because my last big scrape was a nightmare.

full read: https://www.screamingfrog.co.uk/blog/generate-markdown-at-scale/