Project

General

Profile

Bug #255

Java SSL crypto problems

Added by Anil Jangity almost 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Security
Target version:
Start date:
2010-09-24
Due date:
% Done:

0%

Estimated time:
Difficulty:
Tags:

Description

Enabling SSL with (Java) OpenDS fails on b147. The same exact configuration works on OpenSolaris 2009.06 and Linux. Same JDK versions across all platforms.

[23/Sep/2010:16:26:15 -0700] DISCONNECT conn=2 reason="Protocol Error" msg="The client sent a request to the Directory Server that could not be properly decoded as an LDAP message: javax.net.ssl.SSLHandshakeException: Invalid padding"

OpenSSL shows a generic SSL handshake error when connecting to this SSL service behind OpenDS:

anilj@vps1:~$ openssl s_client -connect server:1636
CONNECTED
depth=1 /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
13276:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:../../../../common/openssl/ssl/s23_lib.c:188:
anilj@vps1:~$

If I take the same certificate and use stunnel wrappers, it works.
Is this a bug inside Solaris cryptography framework? It is used by the Java VM from:

/usr/java/jre/lib/security/sunpkcs11-solaris.cfg

java version "1.6.0_18"


Files

SslTestcase.java (588 Bytes) SslTestcase.java Onno Molenkamp, 2010-09-26 11:44 AM

History

#1

Updated by Albert Lee almost 9 years ago

Commenting out the PKCS11 provider from /usr/java/jre/lib/security/java.security should be a workaround for now.

#2

Updated by Onno Molenkamp almost 9 years ago

I've reported this problem a while ago in the OpenSolaris bugtracker, CR 6973705.

As a result of a change introduced in the pkcs11 code in onnv_142, DH key generation is broken. Upstream it's fixed in onnv_150.

I've attached a simple testcase that opens an SSL listener:

keytool -genkeypair -dname cn=test -keystore test.jks -storepass changeit -keypass changeit -keyalg RSA -alias testcert
javac SslTestcase.java
java -Djavax.net.ssl.keyStore=test.jks -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.debug=ssl SslTestcase

Connect to it using openssl:

openssl s_client -quiet -connect 127.0.0.1:12345

This will result in a padding error when run on OpenSolaris >= 142.

#3

Updated by Rich Lowe almost 9 years ago

I'm looking into this from an illumos point of view.

#4

Updated by Albert Lee almost 9 years ago

  • Target version set to oi_148

Tagging this for oi_148 because we almost certainly need to deliver a workaround.

#5

Updated by Rich Lowe almost 9 years ago

This appears to be due to an inadvertent change in buffer size rounding in the diffie-hellman code, though I need to run a few more tests to be sure.

I filed #289 to track this in illumos, leaving this for your workaround (or application of the fix). Sadly, I couldn't copy this issue to anywhere outside the OI project space.

#6

Updated by Albert Lee almost 9 years ago

  • Assignee set to Albert Lee

Thanks to Jason and Rich, we might have a proper fix: http://cr.illumos.org/view/madbjvqy/

#7

Updated by Jason King almost 9 years ago

No might, do :)

Just need to complete a few cosmetic changes (no change in logic) tonight or tomorrow to that patch, and it should be fixed. If you absolutely cannot wait until then, the above patch does resolve the issue.

#8

Updated by Jason King almost 9 years ago

Resolved with changeset 13203. I don't have access to resolve this ticket however.

#9

Updated by Albert Lee almost 9 years ago

  • Status changed from New to In Progress

I committed your updated changes from the webrev this morning to our mq repo, if that differs from the commit I'll redo it when I get home.

#10

Updated by Albert Lee almost 9 years ago

  • Status changed from In Progress to Resolved

Fixed on oi_148.

Also available in: Atom PDF