The straight forward answer is that you cannot tell how the JVM was launched.
But for the example use-case in your question, you don't really need to know how the JVM was launched.  What you really need to know is whether the user will see a message written to the console.  And the way to do that would be something like this:
if (!file.exists()) {
    Console console = System.console();
    if (console != null) {
        console.format("File doesn't exists%n");
    } else if (!GraphicsEnvironment.isHeadless()) {
        JOptionPane.showMessage(null, "File doesn't exists");
    } else {
        // Put it in the log
    }
 }
The javadoc for Console, while not water tight, strongly hints that a Console object (if it exists) writes to a console and cannot be redirected.
Thanks @Stephen Denne for the !GraphicsEnvironment.isHeadless() tip.