0

I have a large csv file with 20.9 GB, it has 16 columns and over 170 million rows.

My computer has 128 GB RAM and Python can use all that is available.

When I try to read just two columns using pandas.read_csv() with low_memory=False I'm getting

ParserError: Error tokenizing data. C error: out of memory

I can read it fine with low_memory=True.

Can someone explain to me why this happens?

Khris
  • 3,132
  • 3
  • 34
  • 54
  • 1
    [this post](https://stackoverflow.com/questions/28219902/pandas-read-csv-on-6-5-gb-file-consumes-more-than-170gb-ram) and [this post](https://stackoverflow.com/questions/24251219/pandas-read-csv-low-memory-and-dtype-options/27232309#27232309) can be of some help as well – MattR Mar 12 '20 at 13:44
  • Thanks for the links. – Khris Mar 12 '20 at 14:10

0 Answers0