0

I have written a cython function using numpy.array to execute some calculation on a big array for example the array has 24000x1500 size. Then I got MemoryError.

I am wondeing what is the ultimate size of a two-dimensional array with float64 components in python?

Is this error occurring due to the size of my array?

What should I do to avoid this error?

Ramsharan
  • 2,054
  • 2
  • 22
  • 26
Dalek
  • 4,168
  • 11
  • 48
  • 100
  • possible duplicate of [Python memory usage of numpy arrays](http://stackoverflow.com/questions/11784329/python-memory-usage-of-numpy-arrays) – André Laszlo Apr 12 '15 at 14:06
  • What is the calculation? An expression involving numpy arrays might result in the creation of several temporary arrays that hold intermediate results. – Warren Weckesser Apr 13 '15 at 01:53

1 Answers1

1

Since a 64 bit float occupies 8 bytes, you can easily estimate a lower bound at 24,000 * 1500 * 8 = 288,000,000 bytes which is about 0.3 GB. I don't know about the overhead created by a numpy array, but I'm guessing it's not significant for an array that size.

According to the answer I linked to in the comment above, you can also find the size using arr.numbytes. It also returns the memory used by the array elements.

I can see how a program using that much memory could throw a MemoryError. Do some memory profiling to find out if it's your big array (likely) or something else that eats your memory. Or buy more memory.

André Laszlo
  • 15,169
  • 3
  • 63
  • 81