Project

General

Profile

Actions
Copy link

Bug #14640

closed

bhyve VNC authentication fails with openssl3

Added by Dan McDonald 4 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Bite-size
Tags:
Gerrit CR:
2030

Description

See https://github.com/omniosorg/illumos-omnios/pull/1157 and https://github.com/omniosorg/illumos-omnios/commit/95a238f2b60dbc52e30864816f8f07d008ff5c55

Basically, VNC authentication uses a 56-bit DES key, presented as 8 bytes: 64 bits, each byte has a parity bit. The default DES_set_key() function checks the parity bits in each DES key byte, and returns failure starting with OpenSSL 3. All OpenSSL versions have a force-parity version DES_set_key_unchecked() (similar to ipseckey(8)'s DES and 3DES key readers, which ignore the input parity bit and recomputes it), and should be used.

Actions
Copy link
 #1

Updated by Robert Mustacchi 4 months ago

  • Project changed from site to illumos gate
Actions
Copy link
 #2

Updated by Andy Fiddaman 4 months ago

  • Subject changed from bhyve VNC authentication fails with openssl3 to bhyve VNC authentication fails with openssl3
Actions
Copy link
 #3

Updated by Electric Monk 4 months ago

  • Gerrit CR set to 2030
Actions
Copy link
 #4

Updated by Andy Fiddaman 4 months ago

I've tested that VNC authentication is working with bhyve linked against openssl 1.0, 1.1 and 3.0 using the MacOS screen sharing client and UltraVNC on Windows.

Prior to this fix it was working if the bhyve binary was linked against 1.0 and 1.1, but not 3.0 since the latter defaults to checking the parity bits of the DES key derived from the authentication password, and these are not set/computed by the VNC server code.

Actions
Copy link
 #5

Updated by Electric Monk 4 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 1aa1f41fe10c6220a7fbef328fac1b72a8355a01

commit  1aa1f41fe10c6220a7fbef328fac1b72a8355a01
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2022-04-23T11:02:39.000Z

    14495 bhyve VNC server should support alternate pixel formats
    14494 bhyve VNC server could support multiple connections
    14505 libumem's umem_update thread could be named
    14506 libidspace does not expose id_allocff()
    14640 bhyve VNC authentication fails with openssl3
    Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
    Reviewed by: Michael Zeller <mike@mikezeller.net>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions
Copy link

Also available in: Atom PDF