Project

General

Profile

Feature #10979

Prefer that SMB change notify not tie up a worker thread

Added by Gordon Ross 5 months ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Start date:
2019-05-14
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

We observe that with a large number of Windows clients performance can be lower than expected because there are not enough SMB worker threads free to process new requests.
One of the main contributing factors to is that SMB change notify requests "tie up" (keep busy) a worker thread for each request. Such requests can and do remain active (blocked) for hours or even days, so the worker thereads assigned to those requests are essentially unavailable for the worker pool during that time.

We should instead use a second taskq job to complete SMB change notify requests, so that for most of the (possibly long) duration such requests are active, they won't need a worker thread.

History

#1

Updated by Gordon Ross 5 months ago

  • Description updated (diff)
  • Status changed from New to In Progress
#2

Updated by Gordon Ross 5 months ago

  • Description updated (diff)
#3

Updated by Gordon Ross 5 months ago

Testing:

Run several change notify requests (each explorer window open in a shared directory will have one) and use mdb to see how many smb requests are change notify with workers associated. i.e.

# mdb -k 
Loading modules: [ ... smbsrv ... nsmb ]
> ::smblist
SERVER           ZONE STATE
ffffff0151ca7ac0 0    RUNNING
  SESSION          IP_ADDR          PORT     DIALECT  STATE
  ffffff0169e59730 10.10.1.69       49254    0x210    NEGOTIATED
    USER             UID   ACCOUNT
    ffffff0169e58e78 1     GWR12NS4\smb
    TREE             TID   SHARE NAME       RESOURCE
    ffffff0158a20060 2     data_one         /volumes/data/one
      OFILE            FID   SMB NODE         CRED
      ffffff0169e53d28 24    ffffff0151ca4688 ffffff01559d55f0
      ffffff0169e53548 27    ffffff0151ca40e8 ffffff01559d55f0
      ffffff0169e532a8 45    ffffff0151ca4688 ffffff01559d55f0
      ODIR             SID   VNODE            PATTERN
      ffffff0169ed4800 0     ffffff0151ca40e8 *
      ffffff0169ed2558 0     ffffff0151ca4688 *
    REQUEST          MSG_ID         WORKER           STATE
    COMMAND
    ffffff014f3f83d0 0x2798         ffffff0005cdbc40 WAITING_EVENT
    smb2_change_notify
>

Before the fix, the change notify requests, all have a worker
as shown by the mdb "::smblist" dcmd.

After the fix, the worker column for most change_notify requests shows zeros.

Fix in production since early 2016

#4

Updated by Electric Monk 5 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit bfe5e737326ea1aafea02849716d8aceacf5c2eb

commit  bfe5e737326ea1aafea02849716d8aceacf5c2eb
Author: Gordon Ross <gwr@nexenta.com>
Date:   2019-05-23T02:59:03.000Z

    10979 Prefer that SMB change notify not tie up a worker thread
    Reviewed by: Kevin Crowe <kevin.crowe@nexenta.com>
    Reviewed by: Matt Barden <Matt.Barden@nexenta.com>
    Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
    Reviewed by: Evan Layton <evan.layton@nexenta.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

Also available in: Atom PDF