Prefer that SMB change notify not tie up a worker thread
We observe that with a large number of Windows clients performance can be lower than expected because there are not enough SMB worker threads free to process new requests.
One of the main contributing factors to is that SMB change notify requests "tie up" (keep busy) a worker thread for each request. Such requests can and do remain active (blocked) for hours or even days, so the worker thereads assigned to those requests are essentially unavailable for the worker pool during that time.
We should instead use a second taskq job to complete SMB change notify requests, so that for most of the (possibly long) duration such requests are active, they won't need a worker thread.
Updated by Gordon Ross 5 months ago
Run several change notify requests (each explorer window open in a shared directory will have one) and use mdb to see how many smb requests are change notify with workers associated. i.e.
# mdb -k Loading modules: [ ... smbsrv ... nsmb ] > ::smblist SERVER ZONE STATE ffffff0151ca7ac0 0 RUNNING SESSION IP_ADDR PORT DIALECT STATE ffffff0169e59730 10.10.1.69 49254 0x210 NEGOTIATED USER UID ACCOUNT ffffff0169e58e78 1 GWR12NS4\smb TREE TID SHARE NAME RESOURCE ffffff0158a20060 2 data_one /volumes/data/one OFILE FID SMB NODE CRED ffffff0169e53d28 24 ffffff0151ca4688 ffffff01559d55f0 ffffff0169e53548 27 ffffff0151ca40e8 ffffff01559d55f0 ffffff0169e532a8 45 ffffff0151ca4688 ffffff01559d55f0 ODIR SID VNODE PATTERN ffffff0169ed4800 0 ffffff0151ca40e8 * ffffff0169ed2558 0 ffffff0151ca4688 * REQUEST MSG_ID WORKER STATE COMMAND ffffff014f3f83d0 0x2798 ffffff0005cdbc40 WAITING_EVENT smb2_change_notify >
Before the fix, the change notify requests, all have a worker
as shown by the mdb "::smblist" dcmd.
After the fix, the worker column for most change_notify requests shows zeros.
Fix in production since early 2016
Updated by Electric Monk 5 months ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
commit bfe5e737326ea1aafea02849716d8aceacf5c2eb Author: Gordon Ross <email@example.com> Date: 2019-05-23T02:59:03.000Z 10979 Prefer that SMB change notify not tie up a worker thread Reviewed by: Kevin Crowe <firstname.lastname@example.org> Reviewed by: Matt Barden <Matt.Barden@nexenta.com> Reviewed by: Rick McNeal <email@example.com> Reviewed by: Evan Layton <firstname.lastname@example.org> Approved by: Dan McDonald <email@example.com>