How should I execute external commands using multithreading in Java?

Question

I want to run an external programs repeated N times, waiting for output each time and process it. Since it's too slow to run sequentially, I tried multithreading. The code looks like this:

public class ThreadsGen {

public static void main(String[] pArgs) throws Exception {
    for (int i =0;i < N ; i++ )
    {
        new TestThread().start();
    }   
}

static class TestThread extends Thread {

public void run() {
        String cmd = "programX";
        String arg = "exArgs"; 

        Process pr;
        try {
            pr = new ProcessBuilder(cmd,arg).start();

        } catch (IOException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        }

        try {
            pr.waitFor();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        //process output files from programX.
        //...
}

However, it seems to me that only one thread is running at a time (by checking CPU usage).
What I want to do is getting all threads (except the one that is waiting for programX to finish) working? What's wrong with my code?

Is it because pr.waitFor(); makes the main thread wait on each subthread?

Answer 1

The waitFor() calls are not your problem here (and are actually causing the spawned Threads to wait on the completion of the spawned external programs rather than the main Thread to wait on the spawned Threads).

There are no guarantees around when Java will start the execution of Threads. It is quite likely, therefore, that if the external program(s) that you are running finish quickly then some of the Threads running them will complete before all the programs are launched.

Also note that CPU usage is not necessarily a good guide to concurrent execution as your Java program is doing nothing but waiting for the external programs to complete. More usefully you could look at the number of executed programs (using ps or Task Manager or whatever).

Answer 2

Isn't yours the same problem as in this thread: How to wait for all threads to finish, using ExecutorService?