Bug #255


Java SSL crypto problems

Added by Anil Jangity over 11 years ago. Updated about 11 years ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:


Enabling SSL with (Java) OpenDS fails on b147. The same exact configuration works on OpenSolaris 2009.06 and Linux. Same JDK versions across all platforms.

[23/Sep/2010:16:26:15 -0700] DISCONNECT conn=2 reason="Protocol Error" msg="The client sent a request to the Directory Server that could not be properly decoded as an LDAP message: Invalid padding"

OpenSSL shows a generic SSL handshake error when connecting to this SSL service behind OpenDS:

anilj@vps1:~$ openssl s_client -connect server:1636
depth=1 /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
13276:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:../../../../common/openssl/ssl/s23_lib.c:188:

If I take the same certificate and use stunnel wrappers, it works.
Is this a bug inside Solaris cryptography framework? It is used by the Java VM from:


java version "1.6.0_18"

Files (588 Bytes) Onno Molenkamp, 2010-09-26 11:44 AM
Actions #1

Updated by Albert Lee over 11 years ago

Commenting out the PKCS11 provider from /usr/java/jre/lib/security/ should be a workaround for now.

Actions #2

Updated by Onno Molenkamp over 11 years ago

I've reported this problem a while ago in the OpenSolaris bugtracker, CR 6973705.

As a result of a change introduced in the pkcs11 code in onnv_142, DH key generation is broken. Upstream it's fixed in onnv_150.

I've attached a simple testcase that opens an SSL listener:

keytool -genkeypair -dname cn=test -keystore test.jks -storepass changeit -keypass changeit -keyalg RSA -alias testcert
java SslTestcase

Connect to it using openssl:

openssl s_client -quiet -connect

This will result in a padding error when run on OpenSolaris >= 142.

Actions #3

Updated by Rich Lowe over 11 years ago

I'm looking into this from an illumos point of view.

Actions #4

Updated by Albert Lee over 11 years ago

  • Target version set to oi_148

Tagging this for oi_148 because we almost certainly need to deliver a workaround.

Actions #5

Updated by Rich Lowe over 11 years ago

This appears to be due to an inadvertent change in buffer size rounding in the diffie-hellman code, though I need to run a few more tests to be sure.

I filed #289 to track this in illumos, leaving this for your workaround (or application of the fix). Sadly, I couldn't copy this issue to anywhere outside the OI project space.

Actions #6

Updated by Albert Lee over 11 years ago

  • Assignee set to Albert Lee

Thanks to Jason and Rich, we might have a proper fix:

Actions #7

Updated by Jason King over 11 years ago

No might, do :)

Just need to complete a few cosmetic changes (no change in logic) tonight or tomorrow to that patch, and it should be fixed. If you absolutely cannot wait until then, the above patch does resolve the issue.

Actions #8

Updated by Jason King over 11 years ago

Resolved with changeset 13203. I don't have access to resolve this ticket however.

Actions #9

Updated by Albert Lee over 11 years ago

  • Status changed from New to In Progress

I committed your updated changes from the webrev this morning to our mq repo, if that differs from the commit I'll redo it when I get home.

Actions #10

Updated by Albert Lee about 11 years ago

  • Status changed from In Progress to Resolved

Fixed on oi_148.


Also available in: Atom PDF