How should I store my numpy arrays in file?

Question

I am running a simulation that creates various arrays and matrices every update cycle that I wish to store. The data is all numerical and ranges from scalars to 3 x 4 matrices. I would ideally save the data to a single file and wish to subsequently analyse the data in pandas. I have tried the csv.DictWriter.writerow() method but this saves the arrays as strings which I cannot do analysis on.

Answer 1

You can use numpy.savez to save multiple numpy arrays into one file.

For analysing you can just load it with numpy.load

Answer 2

Probably the fastest and simplest way of storing and loading your matrix data would be to store them in numpy binary format with np.save (https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.save.html) and then loading it back with numpy load before importing into pandas dataframe.

Binary format will give your speed advantage over any text based formats.

Answer 3

Maybe you can consider the yaml library.

import yaml
import numpy as np

ary = np.zeros((2, 2, 2))

Save to file:

with open('numpy_to.yml', 'w') as outfile:
    yaml.dump(ary, outfile, default_flow_style=False)

Reload data:

with open("numpy_to.yml", 'r') as inputfile:
    ary_back = yaml.load(inputfile)

print(ary_back)

# [[[0. 0.]
#   [0. 0.]]

#  [[0. 0.]
#   [0. 0.]]]

---

If you want to store to a single file, you could dump a dienter code herect:

zeros = np.zeros((2, 2, 2))
ones = np.ones((2, 2, 2))

my_numpies = { 'zeros': zeros, 'ones': ones}