python script to concatenate all the files in the directory into one file

Question

I have written the following script to concatenate all the files in the directory into one single file.

Can this be optimized, in terms of

idiomatic python
time

Here is the snippet:

import time, glob

outfilename = 'all_' + str((int(time.time()))) + ".txt"

filenames = glob.glob('*.txt')

with open(outfilename, 'wb') as outfile:
    for fname in filenames:
        with open(fname, 'r') as readfile:
            infile = readfile.read()
            for line in infile:
                outfile.write(line)
            outfile.write("\n\n")

possible duplicate of [combine multiple text files into one text file using python](http://stackoverflow.com/questions/17749058/combine-multiple-text-files-into-one-text-file-using-python) — llb, Jul 19 '13 at 15:15

Martijn Pieters · Accepted Answer · 2015-06-10T14:43:26.557

49

Use shutil.copyfileobj to copy data:

import shutil

with open(outfilename, 'wb') as outfile:
    for filename in glob.glob('*.txt'):
        if filename == outfilename:
            # don't want to copy the output into the output
            continue
        with open(filename, 'rb') as readfile:
            shutil.copyfileobj(readfile, outfile)

shutil reads from the readfile object in chunks, writing them to the outfile fileobject directly. Do not use readline() or a iteration buffer, since you do not need the overhead of finding line endings.

Use the same mode for both reading and writing; this is especially important when using Python 3; I've used binary mode for both here.

edited Jun 10 '15 at 14:43

answered Jul 19 '13 at 15:11

Martijn Pieters

1,048,767
296
4,058
3,343

Why is important to use the same mode for writing and reading? – Juan David Aug 30 '17 at 15:11
1

@JuanDavid: because shutil will use `.read()` calls on one, `.write()` calls on the other file object, passing the read data from one to the other. If one is open in binary mode, the other in text, you are passing through incompatible data (binary data to a text file, or text data to a binary file). – Martijn Pieters Aug 30 '17 at 18:00
The code here doesn't work with CSV files, dang. But it did give me some good inspiration for how to accomplish this with CSV. I'm relatively new to Python. – brett s Feb 23 '21 at 02:25
@bretts: the contents of the files shouldn't matter; perhaps your CSV files are missing the last newline separator, or are using different delimiter formats? – Martijn Pieters Feb 23 '21 at 09:24

Stephen Miller · Answer 2 · 2015-10-27T11:04:18.623

4

Using Python 2.7, I did some "benchmark" testing of

outfile.write(infile.read())

vs

shutil.copyfileobj(readfile, outfile)

I iterated over 20 .txt files ranging in size from 63 MB to 313 MB with a joint file size of ~ 2.6 GB. In both methods, normal read mode performed better than binary read mode and shutil.copyfileobj was generally faster than outfile.write.

When comparing the worst combination (outfile.write, binary mode) with the best combination (shutil.copyfileobj, normal read mode), the difference was quite significant:

outfile.write, binary mode: 43 seconds, on average.

shutil.copyfileobj, normal mode: 27 seconds, on average.

The outfile had a final size of 2620 MB in normal read mode vs 2578 MB in binary read mode.

edited Oct 27 '15 at 11:04

answered Oct 27 '15 at 10:57

Stephen Miller

503
3
11

Interesting. What platform was that? – ellockie Jun 26 '16 at 06:37
I roughly work on two platforms: Linux Fedora 16, different nodes or Windows 7 Enterprise SP1 with an Intel Core(TM)2 Quad CPU Q9550, 2.83 GHz. I think it was the latter. – Stephen Miller Jul 01 '16 at 13:17

score 3 · Answer 3 · answered Jul 19 '13 at 15:11

3

You can iterate over the lines of a file object directly, without reading the whole thing into memory:

with open(fname, 'r') as readfile:
    for line in readfile:
        outfile.write(line)

answered Jul 19 '13 at 15:11

Brendan Long

53,280
21
146
188

score 2 · Answer 4 · answered Jul 19 '13 at 15:15

2

No need to use that many variables.

with open(outfilename, 'w') as outfile:
    for fname in filenames:
        with open(fname, 'r') as readfile:
            outfile.write(readfile.read() + "\n\n")

answered Jul 19 '13 at 15:15

MGP

2,981
35
34

score 2 · Answer 5 · answered Mar 22 '19 at 08:37

I was curious to check more on performance and I used answers of Martijn Pieters and Stephen Miller.

I tried binary and text modes with shutil and without shutil. I tried to merge 270 files.

Text mode -

def using_shutil_text(outfilename):
    with open(outfilename, 'w') as outfile:
        for filename in glob.glob('*.txt'):
            if filename == outfilename:
                # don't want to copy the output into the output
                continue
            with open(filename, 'r') as readfile:
                shutil.copyfileobj(readfile, outfile)

def without_shutil_text(outfilename):
    with open(outfilename, 'w') as outfile:
        for filename in glob.glob('*.txt'):
            if filename == outfilename:
                # don't want to copy the output into the output
                continue
            with open(filename, 'r') as readfile:
                outfile.write(readfile.read())

Binary mode -

def using_shutil_text(outfilename):
    with open(outfilename, 'wb') as outfile:
        for filename in glob.glob('*.txt'):
            if filename == outfilename:
                # don't want to copy the output into the output
                continue
            with open(filename, 'rb') as readfile:
                shutil.copyfileobj(readfile, outfile)

def without_shutil_text(outfilename):
    with open(outfilename, 'wb') as outfile:
        for filename in glob.glob('*.txt'):
            if filename == outfilename:
                # don't want to copy the output into the output
                continue
            with open(filename, 'rb') as readfile:
                outfile.write(readfile.read())

Running times for binary mode -

Shutil - 20.161773920059204
Normal - 17.327500820159912

Running times for text mode -

Shutil - 20.47757601737976
Normal - 13.718038082122803

Looks like in both modes, shutil performs same while text mode is faster than binary.

OS: Mac OS 10.14 Mojave. Macbook Air 2017.

score 1 · Answer 6 · answered Jul 19 '13 at 15:15

1

The fileinput module provides a natural way to iterate over multiple files

for line in fileinput.input(glob.glob("*.txt")):
    outfile.write(line)

answered Jul 19 '13 at 15:15

iruvar

22,736
7
53
82

1

This would be even better if it didn't confine itself to reading a line at a time. – Marcin Jul 19 '13 at 15:35
@Marcin, that is correct. I used to think this was a cool solution - until I saw Martijn Pieter's `shutil.copyfileobj` humdinger. – iruvar Jul 19 '13 at 15:43

python script to concatenate all the files in the directory into one file

6 Answers6

Linked

Related