GC overhead limit exceeded while read from CSV file

Question

I am using below piece of code for reading the 150MB CSV file and getting GC error

Same code which was causing the problem

    public List<String[]> readCsvFile(String ipFilePath) {
         logger.info("Start executing readCsvFile method !!! on file " + ipFilePath);

         CSVReader csvReader = null;
         List<String[]> allRecrods = null;
         Reader reader = null;
    try {
        reader = Files.newBufferedReader(Paths.get(ipFilePath));
        csvReader = new CSVReader(reader);
        allRecrods = csvReader.readAll();
    } catch (Exception e) {
        logger.error("Error in CsvFileReader !!!");
        e.printStackTrace();
        logger.error("Exception : ", e);
    } finally {
        try {
            reader.close();
            csvReader.close();
        } catch (IOException e) {
            logger.error("Error while closing fileReader/csvFileParser !!!");
            e.printStackTrace();
            logger.error("IOException : ", e);
        }
    }
    return allRecrods;
}

I am getting error on the method : csvReader.readAll() as mentioned above. I am not sure what is the problem which the code, and how to solve this, as the same code is working fine with 20-30 MB files.

Use a [RandomAccessFile](https://docs.oracle.com/javase/7/docs/api/java/io/RandomAccessFile.html) to read from parts of your CSV. Alternatively you can 1) increase the heap space -- Not a good solution though. 2) Split the file; chunk it! — Skynet, Mar 08 '18 at 14:39
Hmm, what about to use standard eg. buffered reader, and then read line by line? That also should work I guess, but ok, maybe will be little strange — xxxvodnikxxx, Mar 08 '18 at 15:14
Possible duplicate of [Error java.lang.OutOfMemoryError: GC overhead limit exceeded](https://stackoverflow.com/questions/1393486/error-java-lang-outofmemoryerror-gc-overhead-limit-exceeded) — the8472, Mar 09 '18 at 20:00

score 1 · Answer 1 · answered Mar 08 '18 at 14:45

1

The simplest solution is to increase heap size with flag "-Xmx" for example: "-Xmx1024m". First you should use some heap size monitoring tool to see if the usage is expected.

answered Mar 08 '18 at 14:45

Guts

748
4
10

StefanE · Answer 2 · 2018-03-09T12:22:52.053

You should not read all the lines but instead process the file line by line and size won't matter and is way more memory efficient.

Example from here: http://www.baeldung.com/java-read-lines-large-file

FileInputStream inputStream = null;
Scanner sc = null;
try {
    inputStream = new FileInputStream(path);
    sc = new Scanner(inputStream, "UTF-8");
    while (sc.hasNextLine()) {
        String line = sc.nextLine();
        // System.out.println(line);
    }
    // note that Scanner suppresses exceptions
    if (sc.ioException() != null) {
        throw sc.ioException();
    }
} finally {
    if (inputStream != null) {
        inputStream.close();
    }
    if (sc != null) {
        sc.close();
    }
}

Edit: Also if you are looking for a framework I can recommend Spring Batch https://projects.spring.io/spring-batch/

GC overhead limit exceeded while read from CSV file

2 Answers2