bhyve VNC authentication fails with openssl3
Basically, VNC authentication uses a 56-bit DES key, presented as 8 bytes: 64 bits, each byte has a parity bit. The default DES_set_key() function checks the parity bits in each DES key byte, and returns failure starting with OpenSSL 3. All OpenSSL versions have a force-parity version DES_set_key_unchecked() (similar to ipseckey(8)'s DES and 3DES key readers, which ignore the input parity bit and recomputes it), and should be used.
Updated by Andy Fiddaman 2 months ago
I've tested that VNC authentication is working with bhyve linked against openssl 1.0, 1.1 and 3.0 using the MacOS screen sharing client and UltraVNC on Windows.
Prior to this fix it was working if the bhyve binary was linked against 1.0 and 1.1, but not 3.0 since the latter defaults to checking the parity bits of the DES key derived from the authentication password, and these are not set/computed by the VNC server code.
Updated by Electric Monk 2 months ago
- Status changed from New to Closed
- % Done changed from 0 to 100
commit 1aa1f41fe10c6220a7fbef328fac1b72a8355a01 Author: Andy Fiddaman <firstname.lastname@example.org> Date: 2022-04-23T11:02:39.000Z 14495 bhyve VNC server should support alternate pixel formats 14494 bhyve VNC server could support multiple connections 14505 libumem's umem_update thread could be named 14506 libidspace does not expose id_allocff() 14640 bhyve VNC authentication fails with openssl3 Reviewed by: Jason King <email@example.com> Reviewed by: Michael Zeller <firstname.lastname@example.org> Approved by: Robert Mustacchi <email@example.com>