illumos gate

We had a customer report some problems with a Java app:

javax.net.ssl.SSLHandshakeException: Could not generate secret
        at java.base/sun.security.ssl.ECDHKeyExchange$ECDHEKAKeyDerivation.t12DeriveKey(ECDHKeyExchange.java:446)
        at java.base/sun.security.ssl.ECDHKeyExchange$ECDHEKAKeyDerivation.deriveKey(ECDHKeyExchange.java:418)
        at java.base/sun.security.ssl.ECDHClientKeyExchange$ECDHEClientKeyExchangeProducer.produce(ECDHClientKeyExchange.java:420)
        at java.base/sun.security.ssl.ClientKeyExchange$ClientKeyExchangeProducer.produce(ClientKeyExchange.java:65)
        at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
        at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:173)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
        at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:716)
        at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:970)
        at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:942)
        at SSLPoke.main(SSLPoke.java:23)
Caused by: java.security.InvalidKeyException: Could not derive key
        at jdk.crypto.cryptoki/sun.security.pkcs11.P11ECDHKeyAgreement.nativeGenerateSecret(P11ECDHKeyAgreement.java:207)
        at jdk.crypto.cryptoki/sun.security.pkcs11.P11ECDHKeyAgreement.engineGenerateSecret(P11ECDHKeyAgreement.java:174)
        at java.base/javax.crypto.KeyAgreement.generateSecret(KeyAgreement.java:660)
        at java.base/sun.security.ssl.ECDHKeyExchange$ECDHEKAKeyDerivation.t12DeriveKey(ECDHKeyExchange.java:431)
        ... 17 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_OBJECT_HANDLE_INVALID
        at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_GetAttributeValue(Native Method)
        at jdk.crypto.cryptoki/sun.security.pkcs11.P11ECDHKeyAgreement.nativeGenerateSecret(P11ECDHKeyAgreement.java:201)
        ... 20 more

Unfortunately, trying to dtrace pkcs11_softtoken.so.1 is challenging due to it being dlopen()ed. However using a provided test program with a modified pkcs11_softtoken.so.1 to spit out some data:

C_GetAttributeValue: sess=272512b13b014035 obj=0
javax.net.ssl.SSLHandshakeException: Could not generate secret
    at java.base/sun.security.ssl.ECDHKeyExchange$ECDHEKAKeyDerivation.t12DeriveKey(ECDHKeyExchange.java:445)

'0' is always an invalid session or object handle. Digging a bit more, the problem is that soft_derivekey() doesn't always set phKey to the created object id. Both the CKM_DH_PKCS_DERIVE and CKM_ECDH1_DERIVE mechanisms in that function skip past the 'common' label (unlike the other mechanisms). That means the *phKey = secret_key->handle; statement isn't always getting executed. The flow in this function is somewhat convoluted, we should try to mildly refactor this to fix this.