Project

General

Profile

Bug #4990

mountd and rpcbind should use libumem(3lib)

Added by Marcel Telka over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
nfs - NFS server and client
Start date:
2014-07-10
Due date:
% Done:

100%

Estimated time:
Difficulty:
Bite-size
Tags:
needs-triage

Description

The malloc(3c) from libc does not scale well for massively multithreaded applications.

It was found that with the stock mountd(1m) we could easily get thousands of mountd threads due to libc malloc(3c) inefficiency. The mountd(1m) becomes very slow, and could exhaust the whole virtual memory.

When the libumem(3lib) was preloaded to mountd, the mountd was able to handle the load well with less than 20 threads.

We should link mountd with libumem(3lib) by default.

The rpcbind(1m) should be linked by default with libumem(3lib) as well for similar reasons.


Related issues

Related to illumos gate - Bug #5214: Limit the number of door server threads in mountd(1m)In Progress2014-10-06

Actions

History

#1

Updated by Marcel Telka over 5 years ago

The issue (and the fix) was found when we tested thousands of NFS clients accessing the NFS server. Without the fix the clients were able to saturate mountd and mountd had to spawn thousands of threads to try to handle the workload. Even with the thousands of mountd threads spawned, we encountered the test failures, because mountd was slow. In addition, the default thread stack size in mountd is 1MB, so we had few cases when mountd (it is 32-bit app) exhausted its virtual memory (4k threads with 1MB stack = 4GB).

Once we preloaded libumem(3lib) to mountd we saw that with the same setup (and same load), the mountd threads started to be so fast that mountd had to spawn only less than 50 threads to handle the load.

We had similar problem with rpcbind and the libumem preload fixed the performance issue there as well.

#2

Updated by Marcel Telka over 5 years ago

  • Status changed from In Progress to Pending RTI
#3

Updated by Electric Monk over 5 years ago

  • Status changed from Pending RTI to Closed
  • % Done changed from 0 to 100

git commit 961519c5bffd5ec670890fc3596d6c4ff1cefea0

commit  961519c5bffd5ec670890fc3596d6c4ff1cefea0
Author: Marcel Telka <marcel.telka@nexenta.com>
Date:   2014-07-11T04:08:47.000Z

    4990 mountd and rpcbind should use libumem(3lib)
    Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
    Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
    Reviewed by: Richard Lowe <richlowe@richlowe.net>
    Reviewed by: Jason King <jason.brian.king@gmail.com>
    Approved by: Dan McDonald <danmcd@omniti.com>

Also available in: Atom PDF