java.lang.OutOfMemoryError: GC overhead limit exceeded evenif if with -Xms1024M -Xmx2048M

Question

I had encountered the problem

Exception in thread “main” java.lang.OutOfMemoryError: GC overhead limit exceeded

I tried to fix this error as a solution is present here

Run->Run Configurations->Click on arguments->inside VM arguments type

-Xms1024M -Xmx2048M

Xms- for minimum limit

Xmx- for maximum limit

then also it showing the same error. what should I do?

current I am parsing 76GB json File. Keep this thing in mind

`-Xms1024M` `-Xmx2048M` mean 1G and 2G and you try to load a 76G file. Even with good GC call, it is not possible imo — Gilles-Antoine Nys, May 17 '18 at 09:43
See following post: https://stackoverflow.com/questions/1393486/error-java-lang-outofmemoryerror-gc-overhead-limit-exceeded — tom, May 17 '18 at 09:48
Can the source file be some how split? Can the parsing lib parse in chunks rather than loading the entire file? Can you use a (much) larger amount of memory for xmx — David, May 17 '18 at 09:49

score 0 · Answer 1 · answered May 17 '18 at 09:48

Agree with @Gilles-Antoine Nys With a 2GB heap if you're trying to load the entire 76GB JSON file into memory you're putting a huge amount of pressure on the collector.

Keep in mind that if the source file is 76GB the in memory representation on the heap in Java will likely be much larger due to overheads of creating objects to represent each node etc and meta data associated with your parsing library.

If you're not loading the entire file at once but parsing you might have more luck but I presume this will depend a lot on how your underlying JSON parsing lib is working.

score 0 · Answer 2 · answered May 17 '18 at 09:50

0

We'd have to see the code that you wrote to parse the file, but this kind of error is usually caused by trying to read the entire file at once instead of using a streaming approach.

Both Jackson and Gson (if you're using a json library, ofcourse) provide streaming functionality.

answered May 17 '18 at 09:50

FrederikVH

476
2
8

Take care that GC closes Stream when overheaded :/ – Gilles-Antoine Nys May 17 '18 at 09:51
@Gilles-AntoineNys True, but I'm not necessarily referring to the java streaming API, but more the general concept of streaming. – FrederikVH May 17 '18 at 10:17

score 0 · Answer 3 · answered May 17 '18 at 09:52

0

You will never be able to process a 76GB of data if you do not stream your file when processing it in Java.

You can stream your file with different solutions :

load your file in a database and request it with your Java code
use a library to stream your file. You can follow advice on this post : JAVA - Best approach to parse huge (extra large) JSON file

answered May 17 '18 at 09:52

Ramzus

46
5

And you can also use Jackson library, as suggest by FrederikVH – Ramzus May 17 '18 at 09:54

score 0 · Accepted Answer · answered May 18 '18 at 06:55

When I was handling Wikipedia dump data(which was more than 50GB)then I have changed VM arguments like this

Run->Run Configurations->Click on arguments->inside VM arguments type

-Xms1g -Xmx8g 

Xms- for minimum limit

Xmx- for maximum limit

It works for me

Note: I have 16GB RAM.

java.lang.OutOfMemoryError: GC overhead limit exceeded evenif if with -Xms1024M -Xmx2048M

4 Answers4