Reading a single file with Multiple Thread: should speed up?

Question

I'm reading a file which conatins 500000 rows. I'm testing to see how multiple thread speed up the process....

private void multiThreadRead(int num){

    for(int i=1; i<= num; i++) { 
        new Thread(readIndivColumn(i),""+i).start(); 
     } 
}

private Runnable readIndivColumn(final int colNum){
    return new Runnable(){
        @Override
        public void run() {
            // TODO Auto-generated method stub
            try {

                long startTime = System.currentTimeMillis();
                System.out.println("From Thread no:"+colNum+" Start time:"+startTime);

                RandomAccessFile raf = new RandomAccessFile("./src/test/test1.csv","r");
                String line = "";
                //System.out.println("From Thread no:"+colNum);

                while((line = raf.readLine()) != null){
                    //System.out.println(line);
                    //System.out.println(StatUtils.getCellValue(line, colNum));
                }


                long elapsedTime = System.currentTimeMillis() - startTime;

                String formattedTime = String.format("%d min, %d sec",  
                        TimeUnit.MILLISECONDS.toMinutes(elapsedTime), 
                        TimeUnit.MILLISECONDS.toSeconds(elapsedTime) -  
                        TimeUnit.MINUTES.toSeconds(TimeUnit.MILLISECONDS.toMinutes(elapsedTime)) 
                    );

                System.out.println("From Thread no:"+colNum+" Finished Time:"+formattedTime);
            } 
            catch (Exception e) {
                // TODO Auto-generated catch block
                System.out.println("From Thread no:"+colNum +"===>"+e.getMessage());

                e.printStackTrace();
            }
        }
    };
}

private void sequentialRead(int num){
    try{
        long startTime = System.currentTimeMillis();
        System.out.println("Start time:"+startTime);

        for(int i =0; i < num; i++){
            RandomAccessFile raf = new RandomAccessFile("./src/test/test1.csv","r");
            String line = "";

            while((line = raf.readLine()) != null){
                //System.out.println(line);
            }               
        }

        long elapsedTime = System.currentTimeMillis() - startTime;

        String formattedTime = String.format("%d min, %d sec",  
                TimeUnit.MILLISECONDS.toMinutes(elapsedTime), 
                TimeUnit.MILLISECONDS.toSeconds(elapsedTime) -  
                TimeUnit.MINUTES.toSeconds(TimeUnit.MILLISECONDS.toMinutes(elapsedTime)) 
            );

        System.out.println("Finished Time:"+formattedTime);
    }
    catch (Exception e) {
        e.printStackTrace();
        // TODO: handle exception
    }

}
    public TesterClass() {

    sequentialRead(1);      
    this.multiThreadRead(1);

}

for num = 1 I get following result:

Start time:1326224619049

Finished Time:2 min, 14 sec

Sequential read ENDS...........

Multi-Thread read starts:

From Thread no:1 Start time:1326224753606

From Thread no:1 Finished Time:2 min, 13 sec

Multi-Thread read ENDS.....

for num = 5 I get following result:

    formatted Time:10 min, 20 sec

Sequential read ENDS...........

Multi-Thread read starts:

From Thread no:1 Start time:1326223509574
From Thread no:3 Start time:1326223509574
From Thread no:4 Start time:1326223509574
From Thread no:5 Start time:1326223509574
From Thread no:2 Start time:1326223509574
From Thread no:4 formatted Time:5 min, 54 sec
From Thread no:2 formatted Time:6 min, 0 sec
From Thread no:3 formatted Time:6 min, 7 sec
From Thread no:5 formatted Time:6 min, 23 sec
From Thread no:1 formatted Time:6 min, 23 sec
Multi-Thread read ENDS.....

My question is: shouldn't multi-threaded read takes approx. 2.13 sec ? Can you please explain why is it taking too long with multi-threaded solution?

Thanks in advance.

possible duplicate of [Java multi-thread application that reads a single file](http://stackoverflow.com/questions/8126277/java-multi-thread-application-that-reads-a-single-file) — Tomasz Nurkiewicz, Jan 10 '12 at 20:16
Threading will not work unless they write to a different disk, in that case both threads are contending to write on to the same file. Hence, threading will not work in that scenario. — , Jan 10 '12 at 20:16
@TomaszNurkiewicz - Not the same, that one was using one thread per file. — Kelly S. French, Jan 10 '12 at 20:22

score 10 · Answer 1 · answered Jan 10 '12 at 20:26

10

The reason you are seeing a slow down when reading in parallel is because the magnetic hard disk head needs to seek the next read position (taking about 5ms) for each thread. Thus, reading with multiple threads effectively bounces the disk between seeks, slowing it down. The only recommended way to read a file from a single disk is to read sequentially with one thread.

answered Jan 10 '12 at 20:26

Tudor

61,523
12
102
142

Thanks for your comments. Even though mutli-threaded reading takes bit more time(eg. 6.2 mins rather than 2.10 mins), still I can save 4/5 mins while I'm iterating 5 times with sequential reading (i.e. sequential read = 10.20 mins; where as 5 threads = 6.20 mins) – Hasan Jan 11 '12 at 13:41

score 9 · Answer 2 · answered Jan 10 '12 at 20:17

9

Since file reading is mainly waiting for disk I/O, you have the problem that the disk won't spin faster just because it's used by many threads :)

answered Jan 10 '12 at 20:17

Joachim Isaksson

176,943
25
281
294

Possibly, the I/O locking strategy would depend both on the java implementation and the underlying operating system. – Joachim Isaksson Jan 10 '12 at 20:26
I am executing a test for the same use case - reading a single file from multiple threads. I found that having multiple threads improves the performance if the underlying storage is a SATA disk drive while improves the performance if its a SAS drive. Would it be because of the point-to-point technology or is my test doing something incorrect? – Andy Dufresne Jun 24 '13 at 13:19

score 2 · Answer 3 · answered Jan 10 '12 at 20:20

Reading from a file is an inherently serial process, assuming no caching, meaning there is a limit to how fast you can retrieve data from a file. Even without file locks (i.e. opening the file read-only) all the threads after the 1st will just block on the disk read so you make all the other threads wait and whichever one is active when the data becomes available is the one that processes the next block.

Reading a single file with Multiple Thread: should speed up?

3 Answers3

Linked