Execute spark-submit programmatically from java

Question

I am trying to execute it via:

Process process = Runtime.getRuntime().exec(spark_cmd);

with no luck. The command ran via shell starts my application which succeeds. Running it via exec start a process which dies shortly after and does nothing. When I try

process.waitFor();

it hangs and waits forever. Real magic begins when I try to read something from the process:

 InputStreamReader isr = new InputStreamReader(process.getErrorStream());
 BufferedReader br = new BufferedReader(isr);

To do so I start a thread that reads from the stream in a while loop:

class ReadingThread extends Thread {

   BufferedReader reader;
   Wontekk(BufferedReader reader) {
      this.reader = reader;

   }

   @Override
   public void run() {
      String line;
      try {
         while ((line = reader.readLine()) != null) {
            System.out.println(line);
         }
      } catch (IOException e) {
         // TODO Auto-generated catch block
         e.printStackTrace();
      }
   }
}

Application starts, does some stuff, and hangs. When I abort my application, spark application wakes up (??????????) and completes remaining work. Does anyone have reasonable explanation of what is happening?

thanks

please look at my answer. Which worked for us.. – Ram Ghadiyaram Sep 12 '16 at 14:16 — Ram Ghadiyaram, Sep 12 '16 at 14:16

score 4 · Answer 1 · edited Jun 02 '23 at 11:14

4

You can send a spark job as spark-submit with the help of Java's SparkLauncher class. Check it out at

https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/launcher/SparkLauncher.html

edited Jun 02 '23 at 11:14

kc2001

5,008
4
51
92

answered Mar 08 '16 at 13:22

Sandeep Purohit

3,652
18
22

: are there any other options as well ? – Ram Ghadiyaram Sep 12 '16 at 18:00

score 0 · Answer 2 · edited May 23 '17 at 12:22

One way is Spark launcher as told by @Sandeep Purohit

I'd offer shell script approach with nohup command to submit job like this...

This worked for me incase of mapreduce executions... same way you can try for spark background jobs as well.

Have a look https://en.wikipedia.org/wiki/Nohup "nohup spark-submit <parameters> 2>&1 < /dev/null &"

When ever, you get messages then you can poll that event and call this shell script. Below is the code snippet to do this...

/**
     * This method is to spark submit
     * <pre> You can call spark-submit or mapreduce job on the fly like this.. by calling shell script... </pre>
     * @param commandToExecute String 
     */
    public static Boolean executeCommand(final String commandToExecute) {
        try {
            final Runtime rt = Runtime.getRuntime();
            // LOG.info("process command -- " + commandToExecute);
            final String[] arr = { "/bin/sh", "-c", commandToExecute};
            final Process proc = rt.exec(arr);
            // LOG.info("process started ");
            final int exitVal = proc.waitFor();
            LOG.trace(" commandToExecute exited with code: " + exitVal);
            proc.destroy();

        } catch (final Exception e) {
            LOG.error("Exception occurred while Launching process : " + e.getMessage());
            return Boolean.FALSE;
        }
             return Boolean.TRUE;
    }

Moreover to debug

ps -aef | grep "your pid or process name"

Below command will list the open files opened by the process..

lsof -p <your process id >

Also, have a look at process.waitFor() never returns

Execute spark-submit programmatically from java

2 Answers2

Linked