Project

General

Profile

Actions

Bug #13722

closed

Enabling SMB3 encryption breaks macOS Big Sur clients

Added by Andrew Stormont 5 months ago. Updated 2 months ago.

Status:
Closed
Priority:
Normal
Category:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Bite-size
Tags:
Gerrit CR:

Description

macOS Big Sur clients are unable to connect to systems that have SMB encryption enabled. Using WireShark I was able to see that they establish a session and after several encrypted messages the traffic stops – then the client attempts to establish a new session. It does this about a dozen times and then connect dialog shudders as if the wrong password had been entered.

With the help of some dtrace and some carefully placed print statements I was able to determine that the client logs off immediately after calling the VALIDATE_NEGOTIATE_INFO ioctl which returns NT_STATUS_ACCESS_DENIED. This appears to be happening because the request is not being signed, which is now a requirement after #11038. When encryption is disabled on the server side macOS signs the requests and everything appears to work okay.

The section in the SMB spec that relates to verifying message signatures (here: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/85df1680-2ee7-4d25-a916-a982371ddc75) begins with: "If Connection.Dialect belongs to the SMB 3.x dialect family and if the decryption in section 3.3.5.2.1.1 succeeds, the server MUST skip the processing in this section." so it seems that this check should be omitted when encryption is enabled, and doing so allows macOS clients to work when encryption is enabled.


Related issues

Related to illumos gate - Bug #13929: SMB encryption does not work with macOS 11.4ClosedAndrew Stormont

Actions
Actions #1

Updated by Electric Monk 5 months ago

  • Gerrit CR set to 1420
Actions #2

Updated by Gordon Ross 5 months ago

The Samba client also exposes this (that's where we first noticed it).
BTW this was exposed after #11038

Actions #3

Updated by Andrew Stormont 5 months ago

With the fix proposed in 1420 the issue is no longer reproducible; mac OS Big Sur clients are able to connect just fine and appear to function normally. I have tested this on two separate illumos systems.

Actions #4

Updated by Electric Monk 5 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 715c0bc682f21743c2b3b52b748c228fbe9524ad

commit  715c0bc682f21743c2b3b52b748c228fbe9524ad
Author: Andrew Stormont <astormont@racktopsystems.com>
Date:   2021-05-03T17:45:20.000Z

    13722 Enabling SMB3 encryption breaks macOS Big Sur clients
    Reviewed by: Jorge Schrauwen <registration@blackdot.be>
    Reviewed by: Gordon Ross <Gordon.W.Ross@gmail.com>
    Reviewed by: Matt Barden <mbarden@tintri.com>

Actions #5

Updated by Cullum Smith 3 months ago

Andrew, can you confirm your setup? I'm running OmniOS r151038g (just released, which supposedly has this fix ) and my MacOS clients still cannot connect to any share with encryption turned on.

Client: MacOS Big Sur 11.4 (intel), connecting via kerberized smb

Server: omnios-r151038-685fb14e78 amd64

smbd/encrypt astring disable
smbd/signing_enabled boolean true
smbd/signing_required boolean true
smbd/encrypt_cipher astring "" 
smbd/max_protocol astring "" 
smbd/min_protocol astring "" 
smbd/domain_member boolean true

Running `zfs set sharesmb=encrypt=enabled tank/myshare` causes MacOS clients to fail mount the share. dmesg on the OmniOS host shows the following:

smbsrv: [ID 211354 kern.warning] WARNING: crypto_decrypt_final failed: 0x48
smbsrv: [ID 656130 kern.warning] WARNING: smb3 decryption failed
Actions #6

Updated by Andrew Stormont 3 months ago

Hi Cullum,

I didn't realise it at the time, but this fix only solves the problem when using the global encryption setting. If you're configuring encryption on a per-share basis you'll need another fix, which I will submit for review shortly.

Actions #7

Updated by Cullum Smith 3 months ago

Thanks for the info! I actually get the same error with `smbd/encrypt = enabled` and `smbd/encrypt = required`.

On the illumos server:

smbsrv: [ID 211354 kern.warning] WARNING: crypto_decrypt_final failed: 0x48
smbsrv: [ID 656130 kern.warning] WARNING: smb3 decryption failed

One the MacOS console:

nbssn_recvhdr: id 86 flags 0x22 Server closed their side of the connection.
nbssn_recvhdr: id 87 flags 0x22 Server closed their side of the connection.
nbssn_recv: nbssn_recvhdr error 32
nbssn_recv: nbssn_recvhdr error 32

I can probably get a pcap if you think that would be useful to you.

Actions #8

Updated by Andrew Stormont 2 months ago

  • Related to Bug #13929: SMB encryption does not work with macOS 11.4 added
Actions #9

Updated by Andrew Stormont 2 months ago

Please see #13929.

Actions

Also available in: Atom PDF