Why do I get "Not enough storage is available to process this command" using Java MappedByteBuffers?

Question

I have a very large array of doubles that I am using a disk-based file and a paging List of MappedByteBuffers to handle, see this question for more background. I am running on Windows XP using Java 1.5.

Here is the key part of my code that does the allocation of the buffers against the file...

try 
{
 // create a random access file and size it so it can hold all our data = the extent x the size of a double
 f = new File(_base_filename);
 _filename = f.getAbsolutePath();
 _ioFile = new RandomAccessFile(f, "rw");
 _ioFile.setLength(_extent * BLOCK_SIZE);
    _ioChannel = _ioFile.getChannel();

    // make enough MappedByteBuffers to handle the whole lot
 _pagesize = bytes_extent;
 long pages = 1;
 long diff = 0;
 while (_pagesize > MAX_PAGE_SIZE)
 {
  _pagesize  /= PAGE_DIVISION;
  pages *= PAGE_DIVISION;

  // make sure we are at double boundaries.  We cannot have a double spanning pages
  diff = _pagesize  % BLOCK_SIZE;
  if (diff != 0) _pagesize  -= diff;

 }

 // what is the difference between the total bytes associated with all the pages and the
 // total overall bytes?  There is a good chance we'll have a few left over because of the
 // rounding down that happens when the page size is halved
 diff = bytes_extent - (_pagesize  * pages);
 if (diff > 0)
 {
  // check whether adding on the remainder to the last page will tip it over the max size
  // if not then we just need to allocate the remainder to the final page
  if (_pagesize  + diff > MAX_PAGE_SIZE)
  {
   // need one more page
   pages++;
  }
 }

 // make the byte buffers and put them on the list
 int size = (int) _pagesize ;  // safe cast because of the loop which drops maxsize below Integer.MAX_INT
 int offset = 0;
 for (int page = 0; page < pages; page++)
 {
  offset = (int) (page * _pagesize );

  // the last page should be just big enough to accommodate any left over odd bytes
  if ((bytes_extent - offset) < _pagesize )
  {
   size = (int) (bytes_extent - offset);
  }

  // map the buffer to the right place 
     MappedByteBuffer buf = _ioChannel.map(FileChannel.MapMode.READ_WRITE, offset, size);

     // stick the buffer on the list
     _bufs.add(buf);
 }

 Controller.g_Logger.info("Created memory map file :" + _filename);
 Controller.g_Logger.info("Using " + _bufs.size() + " MappedByteBuffers");
    _ioChannel.close();
    _ioFile.close(); 
} 
catch (Exception e) 
{
 Controller.g_Logger.error("Error opening memory map file: " + _base_filename);
 Controller.g_Logger.error("Error creating memory map file: " + e.getMessage());
 e.printStackTrace();
 Clear();
    if (_ioChannel != null) _ioChannel.close();
    if (_ioFile != null) _ioFile.close();
 if (f != null) f.delete();
 throw e;
} 

I get the error mentioned in the title after I allocate the second or third buffer.

I thought it was something to do with contiguous memory available, so have tried it with different sizes and numbers of pages, but to no overall benefit.

What exactly does "Not enough storage is available to process this command" mean and what, if anything, can I do about it?

I thought the point of MappedByteBuffers was the ability to be able to handle structures larger than you could fit on the heap, and treat them as if they were in memory.

Any clues?

EDIT:

In response to an answer below (@adsk) I changed my code so I never have more than a single active MappedByteBuffer at any one time. When I refer to a region of the file that is currently unmapped I junk the existing map and create a new one. I still get the same error after about 3 map operations.

The bug quoted with GC not collecting the MappedByteBuffers still seems to be a problem in JDK 1.5.

Answer 1

I thought the point of MappedByteBuffers was the ability to be able to handle structures larger than you could fit on the heap, and treat them as if they were in memory.

No. The idea is / was to allow you to address more than to 2**31 doubles ... on the assumption that you had enough memory, and were using a 64 bit JVM.

(I am assuming that this is a followup question to this question.)

EDIT: Clearly, more explanation is needed.

There are a number of limits that come into play.

1. Java has a fundamental restriction that the length attribute of an array, and array indexes have type int. This, combined with the fact that int is signed and an array cannot have a negative size means that the largest possible array can have 2**31 elements. This restriction applies to 32bit AND 64bit JVMs. It is a fundamental part of the Java language ... like the fact that char values go from 0 to 65535.

2. Using a 32bit JVM places a (theoretical) upper bound of 2**32 on the number of bytes that are addressable by the JVM. This includes, the entire heap, your code, and library classes that you use, the JVM's native code core, memory used for mapped buffers ... everything. (In fact, depending on your platform, the OS may give you considerably less than 2**32 bytes if address space.)

3. The parameters that you give on the java command line determine how much heap memory the JVM will allow your application to use. Memory mapped to using MappedByteBuffer objects does not count towards this.

4. The amount of memory that the OS will give you depends (on Linux/UNIX) on the total amount of swap space configured, the 'process' limits and so on. Similar limits probably apply to Windows. And of course, you can only run a 64bit JVM if the host OS is 64bit capable, and you are using 64bit capable hardware. (If you have a Pentium, you are plain out of luck.)

5. Finally, the amount of physical memory in your system comes into play. In theory, you can ask your JVM to use a heap, etc that is many times bigger than than your machine's physical memory. In practice, this is a bad idea. If you over allocate virtual memory, your system will thrash and application performance will go through the floor.

The take away is this:

• If you use a 32 bit JVM, you probably are limited to somewhere between 2**31 and 2**32 bytes of addressable memory. That's enough space for a MAXIMUM of between 2**29 and 2**30 doubles, whether you use an array or a mapped Buffer.

• If you use a 64 bit JVM, you can represent a single array of 2**31 doubles. The theoretical limit of a mapped Buffer would be 2**63 bytes or 2**61 doubles, but the practical limit would roughly the amount of physical memory your machine has.

Answer 2

When memory mapping a file, it is possible to run out of address space in 32-bit VM. This happens even if the file is mapped in small chunks and those ByteBuffers are no longer reachable. The reason is that GC never kicks in to free the buffers.

Refer the bug at http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6417205