numpy savetxt: save a matrix as row

Question

I'm using numpy savetxt() to save the elements of a matrix to file as a single row (I need to print lots of them in order). This is the method I've found:

import numpy as np

mat = np.array([[1,2,3],
                [4,5,6],
                [7,8,9]])

with open('myfile.dat','a') as handle:
    np.savetxt(handle, mat.reshape(1,mat.size), fmt='%+.8e')
handle.close()

There are 2 questions:

1) Is savetxt() the best option? I need to print 1e5 to 1e7 of these things... and I don't want i/o bottlenecking the actual computation. I'm guessing reopening the file each iteration is a bad plan, speed-wise.

2) Ideally I would have some context data printed to start each row so my output might look like:

(N foo mat):

...
6 -2.309 +1.000 +2.000 ...
7 -4.273 +1.000 +2.000 ...
8 -3.664 +1.000 +2.000 ...
...

I could do this using np.append(), but then the first number won't print as an INT. Is this sort of thing doable directly in savetxt()? Or do I need a C-like fprintf() anyways?

score 0 · Answer 1 · answered Mar 04 '14 at 13:45

0

Pandas has a good to_csv method:

import pandas as pd
import numpy as np

mat = np.array([[1,2,3],
                [4,5,6],
                [7,8,9]])
df = pd.DataFrame(data=mat.astype(np.float))
df.to_csv('myfile.dat', sep=' ', float_format='%+.8e', header=False)

By default it'll add the index (index=True), though if you wanted different context data you could just add that to your data frame and set index=False

$ cat myfile.dat 
0 +1.00000000e+00 +2.00000000e+00 +3.00000000e+00
1 +4.00000000e+00 +5.00000000e+00 +6.00000000e+00
2 +7.00000000e+00 +8.00000000e+00 +9.00000000e+00

answered Mar 04 '14 at 13:45

YXD

31,741
15
75
115

It doesn't look like `to_csv` can append onto an existing file. This would mean I have to carry around a potentially huge data frame in RAM and dump it to file at the end.I guess I can experiment with the feasibility of this... – Paul T. Mar 04 '14 at 15:59
Have a look at http://stackoverflow.com/questions/17530542/how-to-add-pandas-data-to-an-existing-csv-file – YXD Mar 05 '14 at 09:07
I see. For now I'm happy with the `savetxt` solution. It seems sort or roundabout to copy a matrix into a DataFrame just to print it. – Paul T. Mar 05 '14 at 18:18

score 0 · Accepted Answer · answered Mar 05 '14 at 18:11

OK. My original code for printing as an array only works if you want to print once. The mat.reshape() method doesn't just return the reshaped matrix it alters mmat itself. This means the next time through the loop any linalg routines will fail.

To avoid this we need to reshape a copy() of mat. I've also added a tmp variable for clarity.

import numpy as np

mat = np.array([[1,2,3],
                [4,5,6],
                [7,8,9]]) # initialize mat to see format

handle = open('myfile.dat', 'ab')
for n in range(N):
    # perform linalg calculations on mat ...
    meta = foo # based on current mat

    tmp = np.hstack( ([[n]], [[meta]], (mat.copy()).reshape(1,mat.size)) )
    np.savetxt(handle, tmp, fmt='%+.8e')

handle.close()

This gets the context data n and meta in this case. I can live with n being saved as a float.

I did some bench marking to check the i/o cost. I set N=100,000 for the loop, and average the run time for 6 runs:

no i/o, just computations: 9.1 sec
as coded above: 17.2 sec
open 'myfile.dat' to append each iteration: 30.6 sec

So the i/o doubles the runtime and, as expected, constantly opening and closing a file is a bad plan.

numpy savetxt: save a matrix as row

2 Answers2