Relaxing SSL algorithm constrains programmatically

Question

The problem

I'm trying to connect to a local server via a SSL socket. The server uses a SSL certificate that:

1. Is self-signed.
2. Has a MD5withRSA signature (?)

I am aware of the security implications, but for reasons beyond my control, I cannot change or request changes to this certificate, thus I am forced to work with it - or rather, around it.

To overcome the above two points, I have created:

1. A TrustManager that allows all certificates.
2. An AlgorithmConstraints that allows all algorithms.

I end up with the following error:

Caused by: java.security.cert.CertPathValidatorException: Algorithm constraints check failed on signature algorithm: MD5withRSA

It seems that my approach to relaxing the algorithm constrains is flawed.

Note: I am aware that that the MD5withRSA algorithm can be allowed via altering the JVM configuration or a commandline parameter, but I need to do this programmatically.

The code

The TrustManager was created as follows:

TrustManager[] trustManager = new TrustManager[] { 
    new X509TrustManager() {     
        @Override
        public java.security.cert.X509Certificate[] getAcceptedIssuers() { 
            return new X509Certificate[0];
        } 
        @Override
        public void checkClientTrusted( 
            java.security.cert.X509Certificate[] certs, String authType) {
            } 
        @Override
        public void checkServerTrusted( 
            java.security.cert.X509Certificate[] certs, String authType) {
        }
    } 
};

The AlgorithmConstraints was created as follows:

AlgorithmConstraints algorithmConstraints = new AlgorithmConstraints() {
    @Override
    public boolean permits(Set<CryptoPrimitive> primitives, String algorithm, AlgorithmParameters parameters) {
        return true;
    }

    @Override
    public boolean permits(Set<CryptoPrimitive> primitives, Key key) {
        return true;
    }

    @Override
    public boolean permits(Set<CryptoPrimitive> primitives, String algorithm, Key key, AlgorithmParameters parameters) {
        return true;
    }
}

And finally, the socket code is as follows:

SSLContext sslContext = SSLContext.getInstance("TLS"); 

sslContext.init(null, trustManager, new java.security.SecureRandom());

SSLSocketFactory factory = sslContext.getSocketFactory();
SSLSocket sslsocket = (SSLSocket) factory.createSocket(server, port);

SSLParameters sslParameters = new SSLParameters();
sslParameters.setAlgorithmConstraints(algorithmConstraints);

sslsocket.setSSLParameters(sslParameters);

sslSocket.startHandshake();

Suggestions?

Answer 1

It seems that certain algorithms are disabled by default in the JAVA_HOME/lib/jre/security/java.security properties files, including MD5. In particular, the two security properties jdk.tls.disabledAlgorithms and jdk.certpath.disabledAlgorithms appear to be critical.

Thanks to dave_thompson_085 for correcting my confusion about system properties and security properties. By simply using the Security.setProperty() method I was able to connect to an SSL server using a certificate signed with MD5WithRSA. I copies the values of the two security properties from the JAVA_HOME/lib/security/java.security file and removed MD5 from the disabled algorithms, then called Security.setProperty() with the modified list. On my JDK the values of these properties were:

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, DES40_CBC, RC4_40, 3DES_EDE_CBC

jdk.certpath.disabledAlgorithms=MD2, MD5, SHA1 jdkCA & usage TLSServer, \
    RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224

I edited out the MD5 restrictions and executed the following two lines at the beginning of the program before any crypto is used:

Security.setProperty("jdk.tls.disabledAlgorithms", "SSLv3, RC4, DH keySize < 1024, EC keySize < 224, DES40_CBC, RC4_40, 3DES_EDE_CBC");
Security.setProperty("jdk.certpath.disabledAlgorithms", "MD2, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224");

Note that, of course, this leaves you exposed to MD5 vulnerabilities. You only want to be exposed to them for the one self-signed certificate, but now you have to do all the work of disallowing MD5 everywhere else in TLS by yourself.

Answer 2

The solution

As noted in this answer, one needs to use the X509ExtendedTrustManager instead of X509TrustManager. This also negates the need for a custom AlgorithmConstraints implementation, as I have done in the question.

The warning

For whoever finds/uses this solution:

Note that bypassing the standard certificate checking exposes the connection (and by extension the application) to a number of security risks, none of which are minor!

Be sure to implement at least some additional security checks, like certificate pinning, or be at least 100% sure that the application is used in a trusted environment!