H2 tcp over SSL causing javax.net.ssl.SSLHandshakeException

Question

I'm running H2 tcp server.
Java (Servlet 3.1 webapp in Tomcat 8.5):

org.h2.tools.Server tcp = Server.createTcpServer("-ifExists", "-tcpSSL", "-tcpPort", "8089", "-tcpAllowOthers").start();

I'm trying to access it on a remote machine via the H2 Shell on command line:

$ java -cp h2-2.1.210.jar org.h2.tools.Shell

After opening port 8089 on router, tcp connections work:

jdbc:h2:tcp://mySite.net:8089/file:/Users/xyz/tomcat/webapps/ROOT/WEB-INF/database/myh2db;IFEXISTS=TRUE;MODE=MySQL;CASE_INSENSITIVE_IDENTIFIERS=TRUE
jdbc:h2:tcp://mySite.net:8089//Users/xyz/tomcat/webapps/ROOT/WEB-INF/database/myh2db;IFEXISTS=TRUE;MODE=MySQL;CASE_INSENSITIVE_IDENTIFIERS=TRUE

But any attempt at using ssl fails with:

SQL Exception: Connection is broken: "javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

At first I thought it was a Tomcat issue with port 8089, but it doesn't look to be. The router routes incoming requests at its port 80 to Tomcat at port 8080. Tomcat redirects any HTTPS requests Tomcat receives to the router's port 443, and the router routes incoming requests it receives at its port 443 to Tomcat's port 8443.

For H2 tcp SSL (running in Tomcat, bound to port 8089), I have set up the router to route incoming requests at port 8089 to port 8089. So now tcp works, but ssl fails with the javax.net.ssl.SSLHandshakeException.

Tomcat server.xml:

<Connector port="8080"
           protocol="HTTP/1.1"
           connectionTimeout="20000"
           redirectPort="443">
   <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
</Connector>
.
.
.
<Connector port="8443" protocol="org.apache.coyote.http11.Http11Nio2Protocol"
               sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
               scheme="https" secure="true"
               maxThreads="150" SSLEnabled="true" >
     <SSLHostConfig certificateVerification="none" sslProtocol="TLS">
         <Certificate certificateKeystoreFile="conf/tomcat.keystore"
                      type="RSA"
                      certificateKeystorePassword="<password>" />
     </SSLHostConfig>
</Connector>

Any idea what the problem is? How to fix it?
Thanks.

UPDATE 2022-03-28
Thanks to Evgenij's pointer, I did some more research.
I used openssl (on macOS) to create new self-signed certificate. I have the following files from this process:

• server.key (private key)
• server.crt (the certificate)
• server.pem (concatenation of above 2; used to make keystore.pkcs12; otherwise looks useless)
• keystore.pkcs12

I then set six java system properties (keyStore and trustStore for the TcpServer (for anonymous TLS connection from client side)):

• javax.net.ssl.keyStore=/path/to/keystore.pkcs12
• javax.net.ssl.keyStorePassword=<password>
• javax.net.ssl.keyStoreType=pkcs12
• javax.net.ssl.trustStore=/path/to/keystore.pkcs12
• javax.net.ssl.trustStorePassword=<password>
• javax.net.ssl.trustStoreType=pkcs12

Now when I start up Tomcat, I get:

Error starting H2 server
org.h2.jdbc.JdbcSQLNonTransientException: IO Exception: "java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)"; "port: 8089 ssl: true" [90031-210]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:573)
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:496)
    at org.h2.message.DbException.get(DbException.java:216)
    at org.h2.message.DbException.convertIOException(DbException.java:461)
    at org.h2.util.NetUtils.createServerSocketTry(NetUtils.java:214)
    at org.h2.util.NetUtils.createServerSocket(NetUtils.java:177)
    at org.h2.server.TcpServer.start(TcpServer.java:236)
    at org.h2.tools.Server.start(Server.java:521)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4763)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5232)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:727)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1016)
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1903)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
    at java.base/javax.net.ssl.DefaultSSLServerSocketFactory.throwException(SSLServerSocketFactory.java:175)
    at java.base/javax.net.ssl.DefaultSSLServerSocketFactory.createServerSocket(SSLServerSocketFactory.java:188)
    at org.h2.security.CipherFactory.createServerSocket(CipherFactory.java:152)
    at org.h2.util.NetUtils.createServerSocketTry(NetUtils.java:204)
    ... 17 common frames omitted
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
    at java.base/java.security.Provider$Service.newInstance(Provider.java:1901)
    at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
    at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
    at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)
    at java.base/javax.net.ssl.SSLContext.getDefault(SSLContext.java:99)
    at java.base/javax.net.ssl.SSLServerSocketFactory.getDefault(SSLServerSocketFactory.java:114)
    at org.h2.security.CipherFactory.createServerSocket(CipherFactory.java:149)
    ... 18 common frames omitted
Caused by: java.security.KeyStoreException: problem accessing trust store
    at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73)
    at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:278)
    at java.base/sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getTrustManagers(SSLContextImpl.java:1053)
    at java.base/sun.security.ssl.SSLContextImpl$DefaultManagersHolder.<clinit>(SSLContextImpl.java:1023)
    at java.base/sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:1198)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base/java.security.Provider.newInstanceUtil(Provider.java:154)
    at java.base/java.security.Provider$Service.newInstance(Provider.java:1894)
    at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
    at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
    at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)
    at java.base/javax.net.ssl.SSLContext.getDefault(SSLContext.java:99)
    at java.base/javax.net.ssl.SSLServerSocketFactory.getDefault(SSLServerSocketFactory.java:114)
    at org.h2.security.CipherFactory.createServerSocket(CipherFactory.java:149)
    at org.h2.util.NetUtils.createServerSocketTry(NetUtils.java:204)
    at org.h2.util.NetUtils.createServerSocket(NetUtils.java:174)
    ... 16 common frames omitted
Caused by: java.io.IOException: toDerInputStream rejects tag type 45
    at java.base/sun.security.util.DerValue.toDerInputStream(DerValue.java:858)
    at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1974)
    at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
    at java.base/java.security.KeyStore.load(KeyStore.java:1479)
    at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:365)
    at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:313)
    at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:55)
    at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49)
    ... 34 common frames omitted

UPDATE 2022-03-29
@dave_thompson_085, to your comment re toDerInputStream rejects tag type 45, here are the first few lines of the od -tx1 output on the keystore.pkcs12:

$ od -tx1 keystore.pkcs12 
0000000    30  82  09  e9  02  01  03  30  82  09  af  06  09  2a  86  48
0000020    86  f7  0d  01  07  01  a0  82  09  a0  04  82  09  9c  30  82
0000040    09  98  30  82  04  4f  06  09  2a  86  48  86  f7  0d  01  07
0000060    06  a0  82  04  40  30  82  04  3c  02  01  00  30  82  04  35
0000100    06  09  2a  86  48  86  f7  0d  01  07  01  30  1c  06  0a  2a
0000120    86  48  86  f7  0d  01  0c  01  06  30  0e  04  08  7c  2f  ef
0000140    9d  c7  1d  83  a8  02  02  08  00  80  82  04  08  5a  ea  2b
0000160    57  b0  b6  ba  58  4c  b5  45  7c  48  28  e0  5c  94  50  aa
0000200    b0  9b  5d  33  b1  94  0b  eb  5e  85  0b  58  83  3c  f0  86
0000220    24  d7  b9  53  12  fa  ab  8b  b1  fe  b4  2b  5c  b7  2c  0a
...

Do I look for 2d (the hex value for the hyphen character)? I don't see any there. The first time any 2d appears is at offset 320:

$ od -tx1 keystore.pkcs12 | grep 2d
0000320    76  6c  82  cb  b4  2f  b2  78  a3  d3  8d  2d  53  36  5e  65
...

Also: I thought I'd try importing the cert (pem file) into $JAVA_HOME/lib/security/cacerts instead of setting those six Java system properties. So I downloaded the pem file via the browser (clicking on the padlock icon in the URL locator box), and I imported it:

$ keytool -importcert -file cert.pem -alias testcert -cacerts

I then listed the cacerts file and confirmed cert.pem was imported into it. But I also noticed that the Keystore type: JKS:

$ keytool -list -cacerts
Enter keystore password:  
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 214 entries
...

Is that JKS type in conflict with the openssl origin of the cert.pem file, which is pkcs12? Should the pem file have been converted first into JKS format before importing into cacerts?
I tested the H2 TcpServer over ssl, and it still throws the same exception as before:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

UPDATE 2022-03-29, #2
I did over again the truststore and keystore per the instructions given at Oracle using Java keytool: Creating a Keystore to Use with JSSE. And I reset all six JVM system properties. I started up Tomcat running the TcpServer, then I tried again connecting to the server remotely via tcp over ssl. Getting the same exceptions.

UPDATE 2022-03-30
I give up getting anonymous TLS connection to work. I am able to connect non-anonymously from the client to the server when I have the client also have the six JVM system properties point to the client's own set of truststore and keystore (client's own credential) when starting the H2 Shell:

$ java -Djavax.net.ssl.keyStore=/path/to/keystore -Djavax.net.ssl.keyStorePassword=<password> -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.trustStore=/path/to/truststore -Djavax.net.ssl.trustStorePassword=<password> -Djavax.net.ssl.trustStoreType=PKCS12 -cp h2-2.1.210.jar org.h2.tools.Shell

What I did learn was that the truststore must have its own file; it cannot share the same file as the keystore even though the files are equivalent to each other. Sharing the same file leads to:

SEVERE [main] org.apache.catalina.core.StandardService.
initInternal Failed to initialize connector [Connector[org.apache.coyote.http11.
Http11Nio2Protocol-8443]]
...
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

The latest set of truststore and keystore files were all generated via Java keytool. Since I am using Java 11, all files are PKCS12. I have had no success with the OpenSSL files (also PKCS12) mentioned at the very beginning.

I have not been able to replicate the toDerInputStream rejects tag type 45 exception.