jps is a good candidate for this, but please note that
To use the jps command-line tool you need to install a JDK.
Otherwise, you can parse the output of another shell_exec call that uses ps with the arguments you want in order to get the running processes: at this point you can check if the process is present.
$search_string = "[j]ava jar myJarProgram.jar";
$running = shell_exec("ps -A -ww | grep '$search_string'");
or similar.
If $running is empty, you can launch the jar.
Another option is to perform everything with a single shell_exec, both with commands concatanation (simple && and ||) or creating a .sh script and shell_executing that.
EDIT:
According to the user comment, the script must work both for Windows and Linux.
You can use the php PHP_OS predefined constant to check if it's Windows or Linux:
if (strtoupper(substr(PHP_OS, 0, 3)) === 'WIN') etc....
If it's Linux, you can use the shell_exec as reported above.
If it's Windows, you can change the shell_exec string using the tasklist Windows command. I don't know it, but there are already dedicated questions and answers like this one.