Concatenate gzipped files with Python, on Windows

Question

Is there a memory-efficient way to concatenate gzipped files, using Python, on Windows, without decompressing them?

According to a comment on this answer, it should be as simple as:

cat file1.gz file2.gz file3.gz > allfiles.gz

but how do I do this with Python, on Windows?

score 8 · Accepted Answer · answered Aug 13 '13 at 12:23

8

Just keep writing to the same file.

with open(..., 'wb') as wfp:
  for fn in filenames:
    with open(fn, 'rb') as rfp:
      shutil.copyfileobj(rfp, wfp)

answered Aug 13 '13 at 12:23

Ignacio Vazquez-Abrams

776,304
153
1,341
1,358

Erik A. Brandstadmoen · Answer 2 · 2020-01-31T16:37:27.893

1

You don't need python to copy many files to one. You can use standard Windows "Copy" for this:

copy file1.gz /b + file2.gz /b + file3.gz /b allfiles.gz

Or, simply:

copy *.gz /b allfiles.gz

But, if you wish to use Python, Ignacio's answer is a better option.

edited Jan 31 '20 at 16:37

answered Aug 13 '13 at 12:24

Erik A. Brandstadmoen

10,430
2
37
55

You forgot the `+`s and the `/b`s. – Ignacio Vazquez-Abrams Aug 13 '13 at 12:28
1

You have an extraneous `+` before `allfiles.gz` that will cause `file1.gz` to get overwritten. "If you omit Destination, the files are combined and stored under the name of the first file in the list." [source](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/copy) – Stefan Jan 30 '20 at 16:20
You are right. Thanks for the attention to detail. Than said, understanding the meaning of /b before and after files is really difficult to understand.. – Erik A. Brandstadmoen Jan 31 '20 at 16:38

Brionius · Answer 3 · 2013-08-13T12:33:28.110

1

If

cat file1.gz file2.gz file3.gz > allfiles.gz

works, then this should work too:

fileList = ['file1.gz', 'file2.gz', 'file3.gz']
destFilename = 'allfiles.gz'

bufferSize = 8  # Adjust this according to how "memory efficient" you need the program to be.

with open(destFilename, 'wb') as destFile:
    for fileName in fileList:
        with open(fileName, 'rb') as sourceFile:
            chunk = True
            while chunk:
                chunk = sourceFile.read(bufferSize)
                destFile.write(chunk)

edited Aug 13 '13 at 12:33

answered Aug 13 '13 at 12:28

Brionius

13,858
3
38
49

This will truncate input files to the size of the buffer. – Ignacio Vazquez-Abrams Aug 13 '13 at 12:29
Whoops, you're right - forgot the while loop to get entire file - edited, thanks. – Brionius Aug 13 '13 at 12:34
The inner loop can be shortened to `for chunk in iter(lambda: sourceFile.read(bufferSize), ''): destFile.write(chunk)`. Also, please bear in mind that PEP 8 recommends names_with_underscores in preference to camelCase. – user4815162342 Aug 13 '13 at 12:39

score 0 · Answer 4 · answered Dec 15 '17 at 04:42

Fortunately, gzipped files can be directly concatenated via the cat CL command, but unfortunately there doesn't seem to be an obvious python command to do this (in the standard library gzip anyways). However, I only looked briefly. There are probably libraries out there to accomplish this.

Nonetheless, a way to accomplish this using the standard library is to call cat using subprocess:

from subprocess import check_call
command = "cat {} {} > {}".format(file1_path, file2_path, output_name)
check_call(command.split())  # Check call takes a list

To generalize this to arbitrary numbers of inputs, you can do:

inputs = ['input1', 'input2', ... 'input9001']
output_name = 'output.gz'

command = "".join(['cat ', '{} ' * len(inputs), '> {out}'])
_call_ = command.format(*inputs, out=output_name).split()

check_call(_call_)

I hope that is helpful to someone.

Concatenate gzipped files with Python, on Windows

4 Answers4

Linked