Project

General

Profile

Bug #4226

mountd(1m) could run out of file descriptors

Added by Marcel Telka about 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
nfs - NFS server and client
Start date:
2013-10-17
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

The main issue

mountd(1m) is running with limit of 256 file descriptors:

# plimit $(pgrep -x mountd)
13377:  /usr/lib/nfs/mountd
   resource              current         maximum
  time(seconds)         unlimited       unlimited
  file(blocks)          unlimited       unlimited
  data(kbytes)          unlimited       unlimited
  stack(kbytes)         10240           unlimited
  coredump(blocks)      unlimited       unlimited
  nofiles(descriptors)  256             65536
  vmemory(kbytes)       unlimited       unlimited
#

In a case there are many NFS clients trying to mount simultaneously we could easily run out of the file descriptors (each TCP connection to mountd would take one file descriptor). If this is the case, then every attempt to open a file in mountd will fail. We encountered failures in get_seconfig() where nfssec.conf is fopen()ed. This lead to AUTH_TOOWEAK responses to clients.

To solve this issue we should increase the file descriptor limit.

Two other minor issues were found in get_seconfig() during the work on this:

Minor issue 1

In a case there are many NFS clients it can happen that a lot of them calls get_seconfig() in mountd concurrently. The get_seconfig() contains critical section; the main reason for the critical section seems to be that the current gettoken() implementation is not MT-safe.

So, we have a lot of threads in get_seconfig(). Every thread opens the /etc/nfssec.conf file and tries to lock the mutex. If we have a lot of such threads waiting to lock the matching_lock mutex, we are just uselessly wasting file descriptors during waiting for the lock.

We are fixing this by moving the fopen() call inside the critical section.

Minor issue 2

The other minor issue is that both nfs_get_qop_name() and get_seconfig() have their own private matching_lock mutex, but both functions calls the gettoken(). This will be fixed by making matching_lock global.

#1

Updated by Marcel Telka about 7 years ago

  • Status changed from In Progress to Pending RTI
#2

Updated by Marcel Telka about 7 years ago

  • Status changed from Pending RTI to Resolved
commit 97adda444bedd8afa322c1d2233957d40bc8e35c
Author: Marcel Telka <marcel.telka@nexenta.com>
Date:   Wed Oct 16 08:24:11 2013 +0200

    4226 mountd(1m) could run out of file descriptors
    Reviewed by: Gordon Ross <gordon.ross@nexenta.com>
    Reviewed by: Jan Kryl <jan.kryl@nexenta.com>
    Approved by: Eric Schrock <eric.schrock@delphix.com>

Also available in: Atom PDF