Project

General

Profile

Bug #6124

NLM server holds vnodes too long

Added by Marcel Telka about 5 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
nfs - NFS server and client
Start date:
2015-08-14
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

The NLM server creates a nlm_vhold structure for every vnode it touches to make sure the vnode is properly held when there are any active locks for the particular vnode. There one nlm_vhold structure allocated for every host/vnode pair, even in a case the number of locks on the particular vnode is higher than one. The nlm_vhold structure is not used to track all of the locks, it's purpose is just to mark a vnode as "touched" and held for the particular host.

Once a lock is released for the particular vnode, the NLM implementation does not free the nlm_vhold structure, since we are not sure whether this lock is the last one (and the vnode is no longer locked by the particular host), or not. To determine this, some expensive tests are needed, so we defer such tests to the nlm_gc() - the NLM garbage collector.

In the nlm_gc() nlm_vhold structures are scanned to see whether they are still needed or not (if not, they are freed). The problem is that the nlm_gc() checks nlm_vhold structs only for some idle hosts. The non-idle hosts are rarely checked for unused nlm_vhold structs and such vholds are never (or too late) released. This causes the particular vnodes are held for too long time, even the files are removed.

Steps to reproduce the bug

On the NFS server create a dataset with a big file and share it:

# zfs create rpool/export/TEST
# zfs set sharenfs=on rpool/export/TEST
# mkfile 100m /export/TEST/bigfile
# chmod 777 /export/TEST/bigfile
# chmod 777 /export/TEST
# ls -la /export/TEST
total 102416
drwxrwxrwx 2 root root         3 aug 13 19:03 .
drwxr-xr-x 4 root sys          4 aug 13 19:03 ..
-rwxrwxrwx 1 root root 104857600 aug 13 19:03 bigfile
# zfs list rpool/export/TEST
NAME                USED  AVAIL  REFER  MOUNTPOINT
rpool/export/TEST   100M  52,9G   100M  /export/TEST
#

On the NFS client do the following (use the attached lock.c compiled using "gcc -Wall -o lock lock.c"):

# mount -o vers=3 SERVER:/export/TEST /mnt
# while true ; do ./lock lock r 0 0 /mnt/testfile ; sleep 5 ; done &
[1] 3744
# ./lock lock r 0 0 /mnt/bigfile
# rm /mnt/bigfile
#

Now, look back at the server:

# ls -la /export/TEST
total 2
drwxrwxrwx 2 root   root   3 aug 13 19:09 .
drwxr-xr-x 4 root   sys    4 aug 13 19:03 ..
-rwxr-xr-x 1 nobody nobody 0 aug 13 19:08 testfile
# zfs list rpool/export/TEST
NAME                USED  AVAIL  REFER  MOUNTPOINT
rpool/export/TEST   100M  52,9G   100M  /export/TEST
#

And you'll see that the space occupied by the now-deleted bigfile is never reclaimed.

Workaround

To see the space reclaimed you need to kill the while loop at the NFS client (please note the while loop is not touching the deleted bigfile, but another small testfile). The other possible way how to see the space reclaimed is to restart the nlockmgr service on the NFS server.


Files

lock.c (3.51 KB) lock.c Marcel Telka, 2015-08-14 03:13 PM

History

#1

Updated by Marcel Telka about 5 years ago

  • Description updated (diff)
#3

Updated by Marcel Telka about 5 years ago

  • Status changed from In Progress to Pending RTI
#4

Updated by Electric Monk about 5 years ago

  • Status changed from Pending RTI to Closed
  • % Done changed from 0 to 100

git commit b1087aec0c7377254423f3429c35960b116e8b34

commit  b1087aec0c7377254423f3429c35960b116e8b34
Author: Marcel Telka <marcel.telka@nexenta.com>
Date:   2015-08-25T21:17:15.000Z

    6124 NLM server holds vnodes too long
    Reviewed by: Alek Pinchuk <alek.pinchuk@nexenta.com>
    Reviewed by: Dan Fields <dan.fields@nexenta.com>
    Reviewed by: Paul Dagnelie <pcd@delphix.com>
    Reviewed by: Gordon Ross <Gordon.W.Ross@gmail.com>
    Approved by: Dan McDonald <danmcd@omniti.com>

Also available in: Atom PDF