how can I save a Pandas dataframe into a compressed csv file?

Question

I load a very large csv file in a gz format in Pandas 0.18 using

pd=pd.read_csv('myfile.gz')

Without surprise, once the csv is unzipped and loaded into the RAM, it takes a lot of space. However, I still need to save my file after I perform my computations.

Can to_csv store my dataframe in a gz format? The only related question I found on StackOverflow is 3 year old...

score 11 · Accepted Answer · answered Mar 28 '16 at 00:05

11

You could use the parameter compression='gzip'

source

answered Mar 28 '16 at 00:05

Till

4,183
3
16
18

2

When you directly upload to s3, the compression will fail – Shivangi Singh Mar 16 '21 at 18:14
@ShivangiSingh So how to deal with that issue? Could you make it work for s3 bucket? – Tejas Krishna Reddy Aug 03 '21 at 16:13
You can use tempfiles – Shivangi Singh Aug 03 '21 at 16:30

how can I save a Pandas dataframe into a compressed csv file?

1 Answers1

Related