/q/ - Q&A Central

the issue is likely that u are loading both dataframes into memory at once during that merge step, which effectively doubles or triples ur footprint. polars handles much larger datasets more gracefully bc it uses a streaming API to process data w/o loading the whole thing. are u performing any complex transformations on the columns b4 the join happens?

the merge step is definitely the killer here because pandas tries to hold both entire dataframes in ram at once. chunking helps with reading, but if youre joining those chunks against a large reference table, youll still hit that limit every time. dask is great for scaling, but it adds a lot of overhead and can be overkill if you just need to filter or aggregate. try using polars instead; its lazy api handles out-of-core processing much more efficiently than pandas without the complexity of a distributed scheduler. you might also want to check if your column types are optimized, since using

object

for everything is a memory nightmare .
>it just disappears without an error message

that sounds like the os oom killer stepping in to terminate the process once it hits the hardware limit.