Java SSL crypto problems
Enabling SSL with (Java) OpenDS fails on b147. The same exact configuration works on OpenSolaris 2009.06 and Linux. Same JDK versions across all platforms.
[23/Sep/2010:16:26:15 -0700] DISCONNECT conn=2 reason="Protocol Error" msg="The client sent a request to the Directory Server that could not be properly decoded as an LDAP message: javax.net.ssl.SSLHandshakeException: Invalid padding"
OpenSSL shows a generic SSL handshake error when connecting to this SSL service behind OpenDS:
anilj@vps1:~$ openssl s_client -connect server:1636
depth=1 /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
verify error:num=19:self signed certificate in certificate chain
13276:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:../../../../common/openssl/ssl/s23_lib.c:188:
If I take the same certificate and use stunnel wrappers, it works.
Is this a bug inside Solaris cryptography framework? It is used by the Java VM from:
java version "1.6.0_18"
Updated by Onno Molenkamp almost 10 years ago
I've reported this problem a while ago in the OpenSolaris bugtracker, CR 6973705.
As a result of a change introduced in the pkcs11 code in onnv_142, DH key generation is broken. Upstream it's fixed in onnv_150.
I've attached a simple testcase that opens an SSL listener:
keytool -genkeypair -dname cn=test -keystore test.jks -storepass changeit -keypass changeit -keyalg RSA -alias testcert
java -Djavax.net.ssl.keyStore=test.jks -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.debug=ssl SslTestcase
Connect to it using openssl:
openssl s_client -quiet -connect 127.0.0.1:12345
This will result in a padding error when run on OpenSolaris >= 142.
Updated by Rich Lowe almost 10 years ago
This appears to be due to an inadvertent change in buffer size rounding in the diffie-hellman code, though I need to run a few more tests to be sure.
I filed #289 to track this in illumos, leaving this for your workaround (or application of the fix). Sadly, I couldn't copy this issue to anywhere outside the OI project space.