to_csv append mode is not appending to next new line

Question

I have a csv called test.csv that looks like:

accuracy    threshold   trainingLabels
abc         0.506       15000
eew         18.12       15000

And then a dataframe called summaryDF that looks like:

accuracy    threshold   trainingLabels
def         0.116       342
dja         0.121       1271

I am doing:

try:
        if os.stat('test.csv').st_size > 0:
                summaryDF.to_csv(path_or_buf=f, encoding='utf-8', mode='a', header=False)
                f.close()
        else:
            print "empty file"
            with open('test.csv', 'w+') as f:
                summaryDF.to_csv(path_or_buf=f, encoding='utf-8')
                f.close()
    except OSError:
        print "No file"
        with open('test.csv', 'w+') as f:
            summaryDF.to_csv(path_or_buf=f, encoding='utf-8')
            f.close()

Because I want my file to be:

accuracy    threshold   trainingLabels
abc         0.506       15000
eew         18.12       15000
def         0.116       342
dja         0.121       1271

Instead, it is:

accuracy    threshold   trainingLabels
abc         0.506       15000
eew         18.12       15000def            0.116       342   
dja         0.121       1271

How can I solve this? I am guessing using a CSV writer instead of to_csv but clearly the append mode is not skipping the last line of the existing file.

score 3 · Accepted Answer · answered Aug 18 '16 at 12:11

Are you using the pandas package? You do not mention that anywhere.

Pandas does not automatically append a new line, and I am not sure how to force it. But you can just do:

f.write('\n')
summaryDF.to_csv(path_or_buf=f, mode='a', ...)

An unrelated bug in your code:

You seem to have a global file object called f.

When you do this:

with open('test.csv', 'w+') as f:
    ...
    f.close()

The file that you are closing there is the file that you just opened in the with block. You are not closing the global file f because the variable was overshadowed by the f in that scope.

Is this what you want? Either way, it makes no sense. The reason why we use the with scope is to avoid having to close the file explicitly.

You either use:

f = open('filename')
...
f.close()

OR

with open('filename') as f:
    ...

You do not close a file opened within a with block. Using a with block has the additional advantage that the file gets closed even if an exception is raised and the following code is not executed.

to_csv append mode is not appending to next new line

1 Answers1

Linked