Bug #13722
closed
Enabling SMB3 encryption breaks macOS Big Sur clients
100%
Description
macOS Big Sur clients are unable to connect to systems that have SMB encryption enabled. Using WireShark I was able to see that they establish a session and after several encrypted messages the traffic stops – then the client attempts to establish a new session. It does this about a dozen times and then connect dialog shudders as if the wrong password had been entered.
With the help of some dtrace and some carefully placed print statements I was able to determine that the client logs off immediately after calling the VALIDATE_NEGOTIATE_INFO ioctl which returns NT_STATUS_ACCESS_DENIED. This appears to be happening because the request is not being signed, which is now a requirement after #11038. When encryption is disabled on the server side macOS signs the requests and everything appears to work okay.
The section in the SMB spec that relates to verifying message signatures (here: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/85df1680-2ee7-4d25-a916-a982371ddc75) begins with: "If Connection.Dialect belongs to the SMB 3.x dialect family and if the decryption in section 3.3.5.2.1.1 succeeds, the server MUST skip the processing in this section." so it seems that this check should be omitted when encryption is enabled, and doing so allows macOS clients to work when encryption is enabled.
Related issues
Updated by Gordon Ross almost 2 years ago
The Samba client also exposes this (that's where we first noticed it).
BTW this was exposed after #11038
Updated by Andrew Stormont almost 2 years ago
With the fix proposed in 1420 the issue is no longer reproducible; mac OS Big Sur clients are able to connect just fine and appear to function normally. I have tested this on two separate illumos systems.
Updated by Electric Monk almost 2 years ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
git commit 715c0bc682f21743c2b3b52b748c228fbe9524ad
commit 715c0bc682f21743c2b3b52b748c228fbe9524ad
Author: Andrew Stormont <astormont@racktopsystems.com>
Date: 2021-05-03T17:45:20.000Z
13722 Enabling SMB3 encryption breaks macOS Big Sur clients
Reviewed by: Jorge Schrauwen <registration@blackdot.be>
Reviewed by: Gordon Ross <Gordon.W.Ross@gmail.com>
Reviewed by: Matt Barden <mbarden@tintri.com>
Updated by Cullum Smith almost 2 years ago
Andrew, can you confirm your setup? I'm running OmniOS r151038g (just released, which supposedly has this fix ) and my MacOS clients still cannot connect to any share with encryption turned on.
Client: MacOS Big Sur 11.4 (intel), connecting via kerberized smb
Server: omnios-r151038-685fb14e78 amd64
smbd/encrypt astring disable smbd/signing_enabled boolean true smbd/signing_required boolean true smbd/encrypt_cipher astring "" smbd/max_protocol astring "" smbd/min_protocol astring "" smbd/domain_member boolean true
Running `zfs set sharesmb=encrypt=enabled tank/myshare` causes MacOS clients to fail mount the share. dmesg on the OmniOS host shows the following:
smbsrv: [ID 211354 kern.warning] WARNING: crypto_decrypt_final failed: 0x48 smbsrv: [ID 656130 kern.warning] WARNING: smb3 decryption failed
Updated by Andrew Stormont almost 2 years ago
Hi Cullum,
I didn't realise it at the time, but this fix only solves the problem when using the global encryption setting. If you're configuring encryption on a per-share basis you'll need another fix, which I will submit for review shortly.
Updated by Cullum Smith almost 2 years ago
Thanks for the info! I actually get the same error with `smbd/encrypt = enabled` and `smbd/encrypt = required`.
On the illumos server:
smbsrv: [ID 211354 kern.warning] WARNING: crypto_decrypt_final failed: 0x48 smbsrv: [ID 656130 kern.warning] WARNING: smb3 decryption failed
One the MacOS console:
nbssn_recvhdr: id 86 flags 0x22 Server closed their side of the connection. nbssn_recvhdr: id 87 flags 0x22 Server closed their side of the connection. nbssn_recv: nbssn_recvhdr error 32 nbssn_recv: nbssn_recvhdr error 32
I can probably get a pcap if you think that would be useful to you.
Updated by Andrew Stormont over 1 year ago
- Related to Bug #13929: SMB encryption does not work with macOS 11.4 added