Bug #13722


Enabling SMB3 encryption breaks macOS Big Sur clients

Added by Andrew Stormont 23 days ago. Updated 4 days ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


macOS Big Sur clients are unable to connect to systems that have SMB encryption enabled. Using WireShark I was able to see that they establish a session and after several encrypted messages the traffic stops – then the client attempts to establish a new session. It does this about a dozen times and then connect dialog shudders as if the wrong password had been entered.

With the help of some dtrace and some carefully placed print statements I was able to determine that the client logs off immediately after calling the VALIDATE_NEGOTIATE_INFO ioctl which returns NT_STATUS_ACCESS_DENIED. This appears to be happening because the request is not being signed, which is now a requirement after #11038. When encryption is disabled on the server side macOS signs the requests and everything appears to work okay.

The section in the SMB spec that relates to verifying message signatures (here: begins with: "If Connection.Dialect belongs to the SMB 3.x dialect family and if the decryption in section succeeds, the server MUST skip the processing in this section." so it seems that this check should be omitted when encryption is enabled, and doing so allows macOS clients to work when encryption is enabled.

Actions #1

Updated by Electric Monk 23 days ago

  • Gerrit CR set to 1420
Actions #2

Updated by Gordon Ross 11 days ago

The Samba client also exposes this (that's where we first noticed it).
BTW this was exposed after #11038

Actions #3

Updated by Andrew Stormont 7 days ago

With the fix proposed in 1420 the issue is no longer reproducible; mac OS Big Sur clients are able to connect just fine and appear to function normally. I have tested this on two separate illumos systems.

Actions #4

Updated by Electric Monk 4 days ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 715c0bc682f21743c2b3b52b748c228fbe9524ad

commit  715c0bc682f21743c2b3b52b748c228fbe9524ad
Author: Andrew Stormont <>
Date:   2021-05-03T17:45:20.000Z

    13722 Enabling SMB3 encryption breaks macOS Big Sur clients
    Reviewed by: Jorge Schrauwen <>
    Reviewed by: Gordon Ross <>
    Reviewed by: Matt Barden <>


Also available in: Atom PDF