@mdma's answer is correct, you can plug in your own Authenticator to handle authentication, so that there's no popup.
If you're already handling authentication in another way (such as by connection.setRequestProperty("Authorization", ...), as this answer to another question describes), you can use Authenticator.setDefault() to choose to not to use any Authenticator for authentication:
Authenticator.setDefault(null);
This removes the old default Authenticator, so that if your authentication is wrong, you get the error response code via your URLConnection, without opening any popup.
An equivalent way is to set the default to an Authenticator which returns null for getPasswordAuthentication() (which is the default implementation), as in the following code:
Authenticator.setDefault(new Authenticator() { });
But unless you're going to add code to your Authenticator, I don't see a reason to choose this over null.