Enabling SMB3 encryption breaks macOS Big Sur clients
macOS Big Sur clients are unable to connect to systems that have SMB encryption enabled. Using WireShark I was able to see that they establish a session and after several encrypted messages the traffic stops – then the client attempts to establish a new session. It does this about a dozen times and then connect dialog shudders as if the wrong password had been entered.
With the help of some dtrace and some carefully placed print statements I was able to determine that the client logs off immediately after calling the VALIDATE_NEGOTIATE_INFO ioctl which returns NT_STATUS_ACCESS_DENIED. This appears to be happening because the request is not being signed, which is now a requirement after #11038. When encryption is disabled on the server side macOS signs the requests and everything appears to work okay.
The section in the SMB spec that relates to verifying message signatures (here: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/85df1680-2ee7-4d25-a916-a982371ddc75) begins with: "If Connection.Dialect belongs to the SMB 3.x dialect family and if the decryption in section 184.108.40.206.1.1 succeeds, the server MUST skip the processing in this section." so it seems that this check should be omitted when encryption is enabled, and doing so allows macOS clients to work when encryption is enabled.
Updated by Electric Monk 4 days ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
commit 715c0bc682f21743c2b3b52b748c228fbe9524ad Author: Andrew Stormont <email@example.com> Date: 2021-05-03T17:45:20.000Z 13722 Enabling SMB3 encryption breaks macOS Big Sur clients Reviewed by: Jorge Schrauwen <firstname.lastname@example.org> Reviewed by: Gordon Ross <Gordon.W.Ross@gmail.com> Reviewed by: Matt Barden <email@example.com>