Project

General

Profile

Actions
Copy link

Bug #14640

closed

bhyve VNC authentication fails with openssl3

Added by Dan McDonald 4 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Bite-size
Tags:
Gerrit CR:
2030

Description

See https://github.com/omniosorg/illumos-omnios/pull/1157 and https://github.com/omniosorg/illumos-omnios/commit/95a238f2b60dbc52e30864816f8f07d008ff5c55

Basically, VNC authentication uses a 56-bit DES key, presented as 8 bytes: 64 bits, each byte has a parity bit. The default DES_set_key() function checks the parity bits in each DES key byte, and returns failure starting with OpenSSL 3. All OpenSSL versions have a force-parity version DES_set_key_unchecked() (similar to ipseckey(8)'s DES and 3DES key readers, which ignore the input parity bit and recomputes it), and should be used.

Actions
Copy link

Also available in: Atom PDF