the bottleneck isn't always shitty transformations. just saw some stuff about how
databricks disk cache basically pulls parquet or delta files onto worker nodes so you aren't constantly hitting remote storage. it makes sense if you have a bunch of different jobs, like bi refreshes and data science notebooks, all hitting the same curated tables at once. it's basically just local caching for your cloud object storage to stop that latency creep. but obviously it only helps with repeated reads of overlapping data, so it won't fix a single massive one-off scan. also depends on having decent partitioning in place or you're just wasting space. i wonder how much this actually cuts down on infra costs though, or if it's just
magic hiding bad table design. feels like another way to avoid fixing the actual pipeline mess. lol.
full read:
https://dzone.com/articles/databricks-cache-improves-analytics-workloads