2

I have been trying to read data from a large (~270 MB) text file in Java and store it in an ArrayList of Mats and 3D point arrays in order to do Flann-Based Matching. Although these methods work, it takes too much time (roughly 20-30 minutes) to process and store the data. Is there a way to speed up adding the values to the ArrayList?

My text file in the following format: -1.78894 -0.260657 -5.01526 54 15 2899.69 817.542 14 0 0 1 ... (repeats for 124 more ints)

This is my code using BufferedReader, where mDescriptors is an ArrayList of Mat objects and mPoints3d is a 2D ArrayList of 3D points.

public void loadData() {

    // initialize mDescriptors and mPoints3d by iterating through it;
    for(int i = 0; i < 256; i++)
    {
        mDescriptors.add(new Mat());
        mPoints3d.add(new ArrayList<Point3>());
    }

    try {
        // open the database data file and read from it
        Log.i(TAG, "Opening table.txt");
        File fTable = new File(Environment.getExternalStorageDirectory().getPath() + "/GlassProject/table.txt");
        BufferedReader br = new BufferedReader(new FileReader(fTable));

        String str;
        // keep reading data until there is no more data to read
        while((str = br.readLine()) != null)
        {
            String[] sa = str.split(" ");
            Point3 xyz = new Point3(Float.valueOf(sa[0]), Float.valueOf(sa[1]), Float.valueOf(sa[2]));
            int imgidx = Integer.valueOf(sa[3]);
            //int featidx = Integer.valueOf(sa[4]);
            // Iterate through the rest of the line to get the descriptor data (128 values)
            for(int i = 7; i < sa.length; i++)
            {
                mDes.put(0,i - 7,Integer.valueOf(sa[i]));
            }
            mDescriptors.get(imgidx - 1).push_back(mDes); // Index issues happen here
            mPoints3d.get(imgidx - 1).add(xyz);
        }
        Log.v(TAG, "Loaded the Database data!");
    }
    catch(IOException ex) {
        Log.e(TAG, "Error: could not open table");
        ex.printStackTrace();
    }
}
Ken94
  • 21
  • 2
  • Use Traceview and determine where your time is being spent. Note that you may not have enough heap space for what you are trying to read in. – CommonsWare Nov 02 '15 at 20:24
  • After using Traceview and if the file IO is the bottleneck - consider using a compressed file as your source. – Morrison Chang Nov 02 '15 at 20:28
  • check [here](http://stackoverflow.com/questions/14037404/java-read-large-text-file-with-70million-line-of-text), [here](http://stackoverflow.com/questions/2356137/read-large-files-in-java) and [here](http://stackoverflow.com/questions/5868369/how-to-read-a-large-text-file-line-by-line-using-java) (probably one is a duplicate, too) – Miki Nov 02 '15 at 20:52
  • Sorry for the late reply, I found that most of the time was spent trying to add values to the arraylist, especially in that inner for loop. Is there a faster way to add elements to an arraylist? – Ken94 Nov 06 '15 at 18:41
  • What `ArrayList`? Do you mean `mDes`? Where is it declared? Is it created with an iniital capacity of 128? Have you considered using an actual array? – user207421 Aug 11 '16 at 01:55

0 Answers0