Project

General

Profile

Actions

Bug #12243

closed

too many NFS threads actually hurts performance

Added by Gordon Ross over 1 year ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
nfs - NFS server and client
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
nfs-zone
Gerrit CR:

Description

In a previous change ( see #7651 ) the number of NFS server threads was increases from 16 to 1024.
It turns out that in most configurations this actually performs worse than setting max_threads = 256.

We usually see the following when looking at customer systems:

With max number of nfsd threads set too high, those threads are usually stay at the top of the process list. 'prstat' shows CPU percentage utilization for nfsd twice higher than for zpool, storage backend
Thread state inspection shows most of nfsd threads in zio_wait or in zil_commit. Yet ZIO doesn't get adequate amount of CPU cycles to process requests. We are queuing requests in ZIO - while we could instead utilize RAM, to work with listen_backlog(s)
In a nutshell, we would like the storage_backend to process requests as fast as possible, and that means, ZIO shouldn't be starved out of CPU cycles.

servers=1024 would work great on systems with fast dual-socket 16-core Xeons. That's not the case for dual quad- and hexa-core rigs.

We've historically used 256 as the default number of nfsd threads, and that has worked well on the vast majority of systems in the field.

Actions #1

Updated by Dan McDonald over 1 year ago

  • Tags set to nfs-zone
Actions #2

Updated by Dan McDonald over 1 year ago

Testing has been done during the continuing testing of nfs-zone (#11083) where this fix originates.

Actions #3

Updated by Electric Monk over 1 year ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 9f9c12cd25ff2bda305600e1620ad2eecad4ef19

commit  9f9c12cd25ff2bda305600e1620ad2eecad4ef19
Author: Evan Layton <evan.layton@nexenta.com>
Date:   2020-01-24T21:03:53.000Z

    12243 too many NFS threads actually hurts performance
    Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
    Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
    Reviewed by: Dan McDonald <danmcd@joyent.com>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Actions

Also available in: Atom PDF