how are you verifying its a global variable issue? if you arent explicitly appending those chunks to a list outside the loop, pandas should drop the previous chunk from memory once the next iteration starts.
> pd. read_csv('data. csv', chunksize=1000)if youre just doing math or filtering on each chunk and then discarding it, your RAM usage shouldnt climb linearly. check if you are accidentally storing results in a
growing list or a dictionary inside the loop. also, try using
polars with
scan_csv
instead; its lazy evaluation is much better at handling files that exceed physical memory without manual chunking logic ⚠