Is there any way to handle Java heap space exception

Question

I am looking to convert file input stream for big file (The file is of 100MB) and it is throwing and java.lang.OutOfMemoryError : Java Heap space

import java.io.FileInputStream; import java.io.IOException;

import org.apache.commons.io.IOUtils;

public class TestClass {
    public static void main(String args[]) throws IOException
    {
        //Open the input and out files for the streams
        FileInputStream fileInputStream = new FileInputStream("file.pdf");
        IOUtils.toByteArray(fileInputStream);
    } 
}

The actual stack trace is

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:322)
    at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:463)
    at TestClass.main(TestClass.java:12)

I did tried to handle it using the below method

public static byte[] toByteArray(InputStream is) {
        if (is == null) {
            throw new NullPointerException("The InputStream parameter is null.");
        }

        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        try {
            byte[] buffer = new byte[32];
            int read;
            while ((read = is.read(buffer)) != -1) {
                baos.write(buffer, 0, read);
            }
            return baos.toByteArray();
        } catch (IOException e) {

        }

Which then fails with

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2786)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
    at TestClass.toByteArray(TestClass.java:25)
    at TestClass.main(TestClass.java:14)

Is there any way we could handle this !!! Any inputs will be appreciated.

Thanks !!!

Are you sure doing `byte[] buffer = new byte[32];` is a good idea? I recall that when I changed my buffer from 2048 to something less than 128, I was getting serious performance issues. — Water, May 22 '15 at 15:59
Use a profiler to look at your app's memory usage. Objects that are never garbage collected possibly are being held when they could be released. If that fails, buy more memory chips and plug them into your mainboard. — markspace, May 22 '15 at 16:01
Read this article: http://javarevisited.blogspot.com/2011/09/javalangoutofmemoryerror-permgen-space.html — Akash Rajbanshi, May 22 '15 at 16:01
Thanks for the link @Akash , For others , I am working on laptop with 16GB of RAM , it cant be better than that. — Stacker1234, May 22 '15 at 16:04

Kedar Mhaswade · Accepted Answer · 2015-05-22T17:18:09.580

One Oracle java command line flag named -XshowSettings may help you understand what is going on.

On my machine (Ubuntu 12.04) with Oracle JVM 1.7.0_75, this program runs fine, producing no output as desired, for a 100MB file:

$ java -XshowSettings:vm -cp target/classes/:/data/home/kmhaswade/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar foo.TestClass
VM settings:
    Max. Heap Size (Estimated): 1.73G
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

To demonstrate what others have said with respect to JVM's handling of the heap space, I ran it with -Xmx10m and here's what I get (as expected):

$ java -XshowSettings:vm -Xmx10m -cp target/classes/:/data/home/kmhaswade/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar foo.TestClass
VM settings:
    Max. Heap Size: 10.00M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at org.apache.commons.io.output.ByteArrayOutputStream.needNewBuffer(ByteArrayOutputStream.java:122)
        at org.apache.commons.io.output.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
        at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:462)
        at foo.TestClass.main(TestClass.java:12)

Thus, basically, at the line 12 of TestClass, where we are asking for the entire bytearray to be returned, we are running OOM because we started the JVM with maximum 10MB as the heap allocation.

The default max heap in my case is 1.73G because JVM ergonomics determine it based on the class of the machine. Maybe it's different in your case and hence by default it fails for you with a 100MB file.

Is there any way to handle Java heap space exception

1 Answers1

Linked