Java requires larger heap space than what is reasonable

Question

I have a program where I will be using a very large short[] array:

import java.lang.Math;

public class HPTest {

    public static void main(String[] args) {
        int n = 30;
        short[] a = new short[(int)Math.pow(2,n)];
    }
}

As far as I know, a short[] array should use 2 bytes per element, and so an array with 2^30 elements should need about 2 GiB of RAM.

In order to run the program, I therefore tried

java -Xms2000m HPTest

but still got a heap space error. Even at 3000m I got the same error, but at 4000m it worked.

Any ideas as to why I had to go so far above the estimated limit of 2000m?

EDIT: As has been pointed out by many users, I made a very embarrassing error in declaring that a short needs 1 byte rather than 2 bytes. The question then should be why it doesn't suffice with 2000m.

a `byte` is one byte, thus the name. Btw `1 << n` is a more efficient way to calculate 2 to the power of. — Peter Lawrey, Feb 25 '16 at 18:31
@JPMoresmau Whoops, that's rather embarrassing. Still, I need more heap space than what should be necessary. — Alexandre Vandermonde, Feb 25 '16 at 18:32
[The short data type is a 16-bit signed two's complement integer](https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html). Why did you suppose that an array of short values uses 1 byte per element? — skypjack, Feb 25 '16 at 18:32
Are you running 64-bit version of Java? 2Gb of RAM is already too much for 32-bit version. — pelya, Feb 25 '16 at 18:33
When you create a really large array, it doesn't resize the memory regions before rejecting it. If you tune the initial size, and the tenured size, it is more likely to work. — Peter Lawrey, Feb 25 '16 at 18:35
@pelya that depends on the OS. On linux/solaris the 32-bit version can have a 3 GB heap. — Peter Lawrey, Feb 25 '16 at 18:35
@skypjack Yes indeed, I made a silly mistake. So actually, I should need 4 bytes per element, right? — Alexandre Vandermonde, Feb 25 '16 at 18:36
For very large blocks of data I would use off heap memory. This means you don't need to use or size the heap precisely. — Peter Lawrey, Feb 25 '16 at 18:37
@AlexandreVandermonde The primitive type short is defined as 2 bytes, neither 1 nor 4. — skypjack, Feb 25 '16 at 18:38
@AlexandreVandermonde you shouldn't need more than 2 bytes, I suggest trying a number of smaller allocations for comparison. — Peter Lawrey, Feb 25 '16 at 18:38
I just tested, and the smallest max heap that worked on my 64-bit machine is 3073m. Keep in mind that 1gb is 1024m, NOT 1000m. Still doesn't make sense though why 2049m isn't enough since it's a 1B entry array of 2-byte shorts... — mellamokb, Feb 25 '16 at 18:39
I think this should answer your question http://www.ibm.com/developerworks/java/library/j-codetoheap/index.html as array will take extra memory for their sizes , flags and metadata — Pragnani, Feb 25 '16 at 18:47

score 5 · Answer 1 · answered Feb 25 '16 at 19:02

5

Something this large, will be much happier outside the heap. You would be better off looking in to NIO and using direct byte buffers to back your large Short array. This memory can be kept out of the heap, and away from the mitts of the garbage collector (who may at times feel inclined to want to copy your buffer from one area to the other).

See java.nio.ShortBuffer and start digging from there.

answered Feb 25 '16 at 19:02

Will Hartung

115,893
19
128
203

Thanks! On second thought, I might be able to swtich down to a byte array. Anyway, I will basically be using the array as an array of flags corresponding to different containers of all integers from 0 to 2^30-1. Therefore, as I loop through these integers, I will set the flag at the corresponding index of my short/byte array to a certain value. As far as I can see, I can accomplish this by calling put() on the buffer each time. Would this be very slow? – Alexandre Vandermonde Feb 25 '16 at 19:11
Actually, allocations that are this big goes straight to the "Old Gen" pool, and will not be copied around by the garbage collector. I just empirically tested that to be true, and I seem to remember reading that somewhere, but I don't have a reference at this time. – Andreas Feb 25 '16 at 21:29
@AlexandreVandermonde If they are just flags, you should make it a `boolean` array, and the JVM may squeeze 8 booleans into a single byte. – Andreas Feb 25 '16 at 21:31
@Andreas Could you please elaborate a bit on the "Old Gen" pool. Also, flags may have been an inappropriate term, as my flags are not binary but rather n-ary. – Alexandre Vandermonde Feb 25 '16 at 21:42
@AlexandreVandermonde If you want to learn more about Java memory pools, you could read something like this: [Java Platform, Standard Edition HotSpot Virtual Machine Garbage Collection Tuning Guide](https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/toc.html) – Andreas Feb 25 '16 at 23:40
@Andreas I just tried to allocate a ShortBufer of size 2^30 and I still got the same heap space error as I did when I tried to allocate a short array. Did I miss something? – Alexandre Vandermonde Feb 26 '16 at 00:10
@AlexandreVandermonde I believe Will was talking about a `ShortBuffer` obtained by calling `ByteBuffer.asShortBuffer()`, on a *direct* `ByteBuffer` mapped to a file. – Andreas Feb 26 '16 at 01:23

Java requires larger heap space than what is reasonable

1 Answers1