illumos gate

A filesystem mounted over nfs4 from an OI 151a server on a Linux client becomes unresponsive after running a filebench workload on the client for several minutes. Metadata operations (ls, stat, rm, mkdir) still work, but operations involving file contents (e.g. cat) block indefinitely. After restarting the nfs service on the server and waiting for 2 minutes, the nfs4 mount becomes accessible again.

Attached is a tcpdump trace starting about a second before the nfs4 mount stops responding. From packet 8158 onward,the client and server go into a loop where the client gets an NFS4ERR_STALE_CLIENTID on OPEN, it succesfully renews its clientid, and then immediately gets the same error on OPEN again. This seems to be the same problem as described in http://thread.gmane.org/gmane.linux.nfs/44449 . The conclusion of that thread was that this is not a bug in the Linux client.

This behaviour happens both on physical hardware and in Xen PV domains. The problem does not occur when the client is running OI, or the server is running Linux, or over nfs3.

mdb output¶

Before running the filebench workload, "echo -e '::rfs4_db\n::rfs4_client' | mdb -k" (suggested by Albert Lee) shows:

rfs4_database=ffffff01513b4db0
  debug_flags=00000000   shutdown:      count=0 tables=ffffff01513c66c0
------------------ Table ------------------- Bkt  ------- Indices -------
Address          Name          Flags    Cnt  Cnt  Pointer          Cnt  Max
ffffff01513c66c0 DelegStateID  00000000 0000 2047 ffffff014e6d3d00 0002 0002
ffffff01513c67a0 File          00000000 0000 2047 ffffff014dc6a940 0001 0001
ffffff01513c6880 Lockowner     00000000 0000 2047 ffffff014fbc9400 0002 0002
ffffff01513c6960 LockStateID   00000000 0000 2047 ffffff014e50fec0 0002 0002
ffffff01513c6a40 OpenStateID   00000000 0000 2047 ffffff014bfcc380 0003 0003
ffffff01513c6b20 OpenOwner     00000000 0000 2047 ffffff0149d6fd00 0001 0001
ffffff01513c6c00 ClntIP        00000000 0000 2047 ffffff0149d6e440 0001 0001
ffffff01513c6ce0 Client        00000000 0000 2047 ffffff014e5af2c0 0002 0002
Address          dbe              clientid         confirm_verf     NCnfm unlnk
cp_confirmed     Last Access

When the filesystem hangs, I get:

rfs4_database=ffffff01513b4db0
  debug_flags=00000000   shutdown:      count=0 tables=ffffff01513c66c0
------------------ Table ------------------- Bkt  ------- Indices -------
Address          Name          Flags    Cnt  Cnt  Pointer          Cnt  Max
ffffff01513c66c0 DelegStateID  00000000 0330 2047 ffffff014e6d3d00 0002 0002
ffffff01513c67a0 File          00000000 120906 2047 ffffff014dc6a940 0001 0001
ffffff01513c6880 Lockowner     00000000 0000 2047 ffffff014fbc9400 0002 0002
ffffff01513c6960 LockStateID   00000000 0000 2047 ffffff014e50fec0 0002 0002
ffffff01513c6a40 OpenStateID   00000000 121966 2047 ffffff014bfcc380 0003 0003
ffffff01513c6b20 OpenOwner     00000000 1048576 2047 ffffff0149d6fd00 0001 0001
ffffff01513c6c00 ClntIP        00000000 0000 2047 ffffff0149d6e440 0001 0001
ffffff01513c6ce0 Client        00000000 0001 2047 ffffff014e5af2c0 0002 0002
Address          dbe              clientid         confirm_verf     NCnfm unlnk
cp_confirmed     Last Access
ffffff01527e7770 ffffff01527e7708 664ef4771b 6600000000000000 False False
0                2011 Dec 23 15:02:41

Steps to reproduce the problem¶

On server:

# (Ramdisk-based pool is fastest, but disk-based works too)
ramdiskadm -a tempdisk 256m
zpool create temppool /dev/ramdisk/tempdisk
zfs set sharenfs=rw=$CLIENT,root=$CLIENT temppool

On client:

# I used Debian Squeeze (6.0.3), but I expect other distros will work as well.
# uname -a : Linux basil 2.6.32-5-xen-amd64 #1 SMP Mon Oct 3 07:53:54 UTC 2011 x86_64 GNU/Linux
# dpkg -l nfs-common :  nfs-common  1:1.2.2-4  NFS support files common to client and server

# Get filebench, either from distro, or download:
#  http://sourceforge.net/projects/filebench/files/filebench/filebench-1.4.9.1
#  untar; ./configure && make && make install

mkdir /mnt/temppool
mount -t nfs4 -o rw,sync,hard $SERVER:/temppool /mnt/temppool
# Check that filesystem is writeable
touch /mnt/temppool/foo

# Save nfs4test.f (attached) to a file

for i in $(seq 15); do echo ===== $i $(date); filebench -f nfs4test.f; done
# The NFS4 mount should become unresponsive around iteration 8 or 9...