17

I know there are a ton of numpy memory error topics, so I hope I haven't duplicated anything. I'm trying to create a np array using np.zeros((500000,10000)). This works fine on my Mac with 16G of memory, but on a Linux server with 28G of RAM it fails instantly with Memory Error. I've verified that I'm running the 64 bit version of Ubuntu and Python, and I'm on Numpy 1.9.3. The only difference I noticed between systems (apart from the obvious) is that when running ulimit -a I get:

Linux: max locked memory (kbytes, -l) 64

Mac: max locked memory (kbytes, -l) unlimited

Could this be the reason I can't run this command? If not, is there some other configuration option I'm missing?

jpavs
  • 648
  • 5
  • 17
  • Is the server yours or a host you are just using administered by someone else? – Shawn Mehan Sep 22 '15 at 17:55
  • It's on AWS but I have sudo access – jpavs Sep 22 '15 at 18:01
  • 3
    well, you will want to read something like [this](https://software.intel.com/en-us/blogs/2014/12/16/best-known-methods-for-setting-locked-memory-size). You seem to have been locked down by admin. I don't know if you can affect this yourself or need to ask for the hand of Bezos, but you were on the right track. Good luck. – Shawn Mehan Sep 22 '15 at 18:04
  • 3
    @ShawnMehan This is a few months later, but it turned out that I had no virtual memory when I tried to do this. I had to cajole my sysadmin into giving me like 20GB but it worked after that :) – jpavs Jan 21 '16 at 15:18
  • OS X Mavericks and later also have some new memory management. In my experience, I've found that it allows me to allocate huge chunks of memory that would otherwise cause `malloc` to fail. Not sure if that's related. – sudo Aug 18 '17 at 07:35
  • Does this help? https://gist.github.com/kiyo-masui/1447078/4ade0558d64a42429c65506a8f64afe8acd5d972 – Tarun Lalwani Apr 25 '18 at 15:34

1 Answers1

4

My best guess are:

  1. The Mac has a swap that allows more allocated memory than the RAM you see.
  2. The Mac does not realise that the array does not fit in memory until the memory is actually used. So the array actually does not fit in memory but you will not know it until you use that memory.

I base my first guess in the fact that in 64 bit your array will take 500000*10000*8= 40GB of RAM 20GB in 32 bit, and therefore the array does not fit in the memory you have. There may be a swap to account for the missing memory.

I base my second guess in this link, where it is explained that np.zeros will not allocate actually in memory the zeros until that memory is accessed for the first time. I have tested in my linux (Ubuntu) computer that np.zeros works with increasing arrays until I reach my RAM limit. Then I get a memory error even if it does not actually allocate the memory.

Once you create the matrix (increase the size enough to make it clear the memory usage):

a = np.zeros((50,10))

You can check the actual memory required by storing a zero in each cell of the matrix:

a[:,:] = 0.0

Or forcing an operation so the memory is accessed and therefore allocated:

a = a + a

Keep track of the memory usage of the computer while performing this check to understand when the memory is allocated.

Mantxu
  • 319
  • 2
  • 11