Bug #255
closed
Java SSL crypto problems
0%
Description
Enabling SSL with (Java) OpenDS fails on b147. The same exact configuration works on OpenSolaris 2009.06 and Linux. Same JDK versions across all platforms.
[23/Sep/2010:16:26:15 -0700] DISCONNECT conn=2 reason="Protocol Error" msg="The client sent a request to the Directory Server that could not be properly decoded as an LDAP message: javax.net.ssl.SSLHandshakeException: Invalid padding"
OpenSSL shows a generic SSL handshake error when connecting to this SSL service behind OpenDS:
anilj@vps1:~$ openssl s_client -connect server:1636
CONNECTED
depth=1 /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
13276:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:../../../../common/openssl/ssl/s23_lib.c:188:
anilj@vps1:~$
If I take the same certificate and use stunnel wrappers, it works.
Is this a bug inside Solaris cryptography framework? It is used by the Java VM from:
/usr/java/jre/lib/security/sunpkcs11-solaris.cfg
java version "1.6.0_18"
Files
Updated by Albert Lee over 12 years ago
Commenting out the PKCS11 provider from /usr/java/jre/lib/security/java.security should be a workaround for now.
Updated by Onno Molenkamp over 12 years ago
- File SslTestcase.java SslTestcase.java added
I've reported this problem a while ago in the OpenSolaris bugtracker, CR 6973705.
As a result of a change introduced in the pkcs11 code in onnv_142, DH key generation is broken. Upstream it's fixed in onnv_150.
I've attached a simple testcase that opens an SSL listener:
keytool -genkeypair -dname cn=test -keystore test.jks -storepass changeit -keypass changeit -keyalg RSA -alias testcert
javac SslTestcase.java
java -Djavax.net.ssl.keyStore=test.jks -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.debug=ssl SslTestcase
Connect to it using openssl:
openssl s_client -quiet -connect 127.0.0.1:12345
This will result in a padding error when run on OpenSolaris >= 142.
Updated by Rich Lowe over 12 years ago
I'm looking into this from an illumos point of view.
Updated by Albert Lee over 12 years ago
- Target version set to oi_148
Tagging this for oi_148 because we almost certainly need to deliver a workaround.
Updated by Rich Lowe over 12 years ago
This appears to be due to an inadvertent change in buffer size rounding in the diffie-hellman code, though I need to run a few more tests to be sure.
I filed #289 to track this in illumos, leaving this for your workaround (or application of the fix). Sadly, I couldn't copy this issue to anywhere outside the OI project space.
Updated by Albert Lee over 12 years ago
- Assignee set to Albert Lee
Thanks to Jason and Rich, we might have a proper fix: http://cr.illumos.org/view/madbjvqy/
Updated by Jason King over 12 years ago
No might, do :)
Just need to complete a few cosmetic changes (no change in logic) tonight or tomorrow to that patch, and it should be fixed. If you absolutely cannot wait until then, the above patch does resolve the issue.
Updated by Jason King over 12 years ago
Resolved with changeset 13203. I don't have access to resolve this ticket however.
Updated by Albert Lee over 12 years ago
- Status changed from New to In Progress
I committed your updated changes from the webrev this morning to our mq repo, if that differs from the commit I'll redo it when I get home.
Updated by Albert Lee over 12 years ago
- Status changed from In Progress to Resolved
Fixed on oi_148.