0

I have the following code for reading a file of about 300MB of size and the execution is interrumped by a "java.lang.OutOfMemoryError: Java heap space" exception:

import java.io.*;
import java.util.*;
 
class MainClass {
   static ArrayList<String> getAttribute(String str) {
      if(str==null) return null;      
      ArrayList<String> a = new ArrayList<>();
      int i = 0;
      int j = -1;
      while(i<str.length() && j<str.length()) {
         j = str.indexOf(',',i);
         if(j==-1) j = str.length();   
         a.add(str.substring(i,j));
         i = str.indexOf(',',j)+1;
      }      
      return a;
   }
   
   static ArrayList<ArrayList<String>> readFile(String in) throws IOException {
      ArrayList<ArrayList<String>> a = new ArrayList<>();
      try(FileInputStream fin = new FileInputStream(in);) {         
         BufferedReader br = new BufferedReader(new InputStreamReader(fin));
         String str = br.readLine();
         while(str!=null) {
            a.add(getAttribute(str));
            str = br.readLine(); //line 26
         }
      }
      return a;
   }   
   
   public static void main(String args[]) throws IOException
   {      
      readFile("stats1.csv"); //line 34
   }
}

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at MainClass.readFile(MainClass.java:26)
        at MainClass.main(MainClass.java:34)

What is the cause of this error? or is it just that the file is too big?

gian_
  • 95
  • 6
  • Can you add the complete stacktrace? Which line should cause the problem? – flaxel Sep 29 '20 at 16:55
  • @flaxel OK, I updated with the stracktrace and line numbers. – gian_ Sep 29 '20 at 17:20
  • 1
    Well, you are reading the whole file in memory, that will eventually blow up. Instead load and parse the file line by line. It will probably be faster and the file size suddenly doesn't matter anymore. – M. Deinum Sep 29 '20 at 17:21
  • "What is the cause of this error? or is it just that the file is too big?" Well, are you able to reproduce the error using a small file? – Karl Knechtel Sep 29 '20 at 17:21
  • @M.Deinum Thanks for the response. I was actually asked to do it like that, copying the whole file into one bidimensional ArrayList. – gian_ Sep 29 '20 at 18:13
  • @KarlKnechtel Yes, it works for smaller files. Since I'm still new to Java I wondered if it was a combination of both, big file plus somehow a memory leak, or some other serious issue that I can't think of. – gian_ Sep 29 '20 at 18:18
  • @derpirscher Yes, it helps, thanks. This allows me to make the program work in my computer at least. – gian_ Sep 29 '20 at 19:34

1 Answers1

2

Look at the source code for the default constructor, that takes no arguments, of class ArrayList. It creates an array of size ten. Then look at the source code for method add() in class ArrayList. Once ten elements have been added, the array needs to be increased. It is increased by half. In other words, when you add the eleventh element, the array is increased to size 15. When you add the sixteenth element, the array is increased to 22 and so on. Once you get to big numbers, adding a single element will cause a huge increase in the array size. Hence it is recommended to use the ArrayList constructor that accepts an initial capacity argument. So maybe you could count the number of lines in the file to get an idea of what capacity your ArrayList requires. Refer to the SO question Java NIO Files count() Method for Counting the Number of Lines for one way to count the number of lines in a file.

Abra
  • 19,142
  • 7
  • 29
  • 41