so i was working this out and had that classic moment where my pipeline runs smoothly for small datasets (like the usual test cases) then hits an unexpected wall when scaling up to bigger data. you know, like going from testing with just your friends' photos in ml land… everything seems fine until it's time to run on all 10k of those holiday party pics. turns out my pipeline was choking at one million samples-same model and same hardware but something broke down when the volume increased significantly. i spent a good week trying different approaches before figuring that data pipelines can be tricky beasts, especially as they grow bigger! anyone else hit similar snags scaling up their ml workflows? how'd you tackle it or did your pipeline work just fine at all sizes too? what's been working (or not) for ya in terms of handling larger datasets efficiently with ray and friends? #raydata #mlpipelineproblems
Source:
https://dev.to/mketkar/your-ray-data-pipeline-works-at-10k-samples-heres-why-it-crashes-at-1m-2g7k