np.save for large data in a for loop

Question

As far I know np.save writes the array all at once outside the loop. Is it possible to put np.save inside the loop and save each row appending to a single file. The data I have is very large and if I declare an array of size (200k X 20k), python crashes. I know the shape of the array.

Sample code

for doPr in range(original_file.shape[0]):
    each_row = original_file[doPr, :]
    each_row = each_row + add_something
    np.save(path+"original_mod_file.npy", each_row)

original_file I read from hard drive row per row. original_mod_file.npy is what I want to write row per row and it will be of size (200k X 20k)

Does this answer your question? [Python: writing large array of arrays to text file](https://stackoverflow.com/questions/11589583/python-writing-large-array-of-arrays-to-text-file) — N. Kiefer, Jun 23 '20 at 09:06
Does this answer your question? [save numpy array in append mode](https://stackoverflow.com/questions/30376581/save-numpy-array-in-append-mode) — user7138814, Oct 22 '22 at 21:06

score 1 · Answer 1 · answered Jun 23 '20 at 09:01

Copying from https://numpy.org/doc/stable/reference/generated/numpy.save.html

with open('test.npy', 'wb') as f:
    np.save(f, np.array([1, 2]))
    np.save(f, np.array([1, 3]))

open the file you want to save to and edit the mode ('rb') to 'ab+'. This creates a file if it doesn't exists and let's you append to it. You can then just append your data in a loop.

np.save for large data in a for loop

Sample code

1 Answers1