Bug #6476
Taking exported_lock RW_READER lock in common_dispatch() can cause deadlock in nfssrv
Start date:
2015-11-24
Due date:
% Done:
0%
Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:
Description
RWLOCK(9F)says
"NOTE: It is a programming error for any thread to acquire an rwlock as
RW_READER that it already holds. Doing so can deadlock the system: if
thread R acquires the lock as RW_READER, then thread W tries to acquire
the lock as a writer, W will set write-wanted and block. When R tries
to get its second read hold on the lock, it will honor the write-wanted
bit and block waiting for W; but W cannot run until R drops the lock.
Thus threads R and W deadlock. To opt out of this behavior -- that is,
to safely allow a lock to be grabbed recursively as a reader -- the
lock should be acquired as RW_READER_STARVEWRITER, which will allow R
to get its second read hold without regard for the write-wanted bit set
by W. Note that the RW_READER_STARVEWRITER behavior will starve
writers in the presence of infinite readers; it should be used with
care, and only where the default RW_READER behavior is unacceptable."
Here is an example:
> ::stacks -v -m nfssrv
mdb: stacks: processing kernel threads
mdb: stacks: 128 unique stacks / 3449 threads
mdb: stacks: done
THREAD STATE SOBJ COUNT
ffffff13925ad3c0 SLEEP RWLOCK 296
swtch+0x141
turnstile_block+0x21a
rw_enter_sleep+0x236
checkexport+0x3b
common_dispatch+0x2b2
rfs_dispatch+0x2d
svc_getreq+0x1c1
svc_run+0xe0
svc_do_run+0x8e
nfssys+0x111
_sys_sysenter_post_swapgs+0x153
ffffff13df1a93c0 SLEEP RWLOCK 16
swtch+0x141
turnstile_block+0x21a
rw_enter_sleep+0x236
checkexport+0x3b
nfs_vptoexi+0x168
nfs_getfh+0x10c
stubs_common_code+0x59
nfssys+0x3c5
_sys_sysenter_post_swapgs+0x153
ffffff008b9cac40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
nfsauth_refresh_thread+0x21d
thread_start+8
ffffff13da590be0 SLEEP RWLOCK 1
swtch+0x141
turnstile_block+0x21a
rw_enter_sleep+0x19b
exportfs+0xedb
stubs_common_code+0x59
nfs_export+0x9e
zfs_ioc_share+0x43
zfsdev_ioctl+0x4a7
cdev_ioctl+0x39
spec_ioctl+0x60
fop_ioctl+0x55
ioctl+0x9b
_sys_sysenter_post_swapgs+0x153
ffffff13df08d860 SLEEP RWLOCK 1
swtch+0x141
turnstile_block+0x21a
rw_enter_sleep+0x236
checkexport+0x3b
rfs3_rename+0x117
common_dispatch+0x5a0
rfs_dispatch+0x2d
svc_getreq+0x1c1
svc_run+0xe0
svc_do_run+0x8e
nfssys+0x111
_sys_sysenter_post_swapgs+0x153
> exported_lock::rwlock
ADDR OWNER/COUNT FLAGS WAITERS
ffffffffc01218a8 READERS=1 B011 ffffff13c4d02040 (R)
|| ffffff13c8bcaba0 (R)
WRITE_WANTED -------+| ffffff13dc737080 (R)
HAS_WAITERS --------+ ffffff13e19f54e0 (R)
ffffff13dd516520 (R)
ffffff13c95a2b00 (R)
ffffff13c925a860 (R)
ffffff13d62ec080 (R)
ffffff13d6260120 (R)
ffffff1364d3a020 (R)
ffffff13925a4100 (R)
ffffff13d4a8a480 (R)
ffffff13df0b0740 (R)
ffffff13a9e33ae0 (R)
ffffff13d468d0c0 (R)
ffffff13c8e3f3a0 (R)
ffffff13c8c0ec20 (R)
ffffff13d4b30020 (R)
ffffff13c94aa440 (R)
ffffff13df13e7e0 (R)
ffffff13d62e80a0 (R)
ffffff13db1ed480 (R)
ffffff1408d07c20 (R)
ffffff13c4d48100 (R)
ffffff13d62483a0 (R)
ffffff13e68637c0 (R)
ffffff13e199a4a0 (R)
ffffff13df0503e0 (R)
ffffff13da660b00 (R)
ffffff13d4b37520 (R)
ffffff13c93363c0 (R)
ffffff13d62f3b40 (R)
ffffff13da665740 (R)
ffffff1408cce840 (R)
ffffff13e6863420 (R)
ffffff13df086140 (R)
ffffff13c9435060 (R)
ffffff1408cdeb40 (R)
ffffff13d4a8c460 (R)
ffffff13d5eab860 (R)
ffffff1391f9cc20 (R)
ffffff13d485c740 (R)
ffffff13c8c21bc0 (R)
ffffff13df1b8c40 (R)
ffffff13e684c800 (R)
ffffff13e687d120 (R)
ffffff13dbb60160 (R)
ffffff13d61c3020 (R)
ffffff13d4876860 (R)
ffffff13c925c100 (R)
ffffff13c8c0e880 (R)
ffffff13dc7a6480 (R)
ffffff13db1d17a0 (R)
ffffff13df098820 (R)
ffffff13c59194a0 (R)
ffffff13d46c7440 (R)
ffffff1408ca44e0 (R)
ffffff13d62e8440 (R)
ffffff13c0d26ae0 (R)
ffffff13df09c080 (R)
ffffff13dc724c60 (R)
ffffff13dd518c40 (R)
ffffff13d62ec420 (R)
ffffff13dbcdeb00 (R)
ffffff13da668180 (R)
ffffff13df0b7160 (R)
ffffff13c9327b80 (R)
ffffff13e68d7b00 (R)
ffffff13e5b3a840 (R)
ffffff13d4a8db80 (R)
ffffff1364d1dc00 (R)
ffffff13e26d9520 (R)
ffffff13d48b3440 (R)
ffffff13c8c210e0 (R)
ffffff13e26bbbc0 (R)
ffffff13c58ca060 (R)
ffffff13df1ec880 (R)
ffffff13d4b7eb80 (R)
ffffff1408ccebe0 (R)
ffffff13df13e0a0 (R)
ffffff139246ac40 (R)
ffffff13df091100 (R)
ffffff13d4b78460 (R)
ffffff13d491e120 (R)
ffffff13dc7a9460 (R)
ffffff137dda3520 (R)
ffffff13e26a1c40 (R)
ffffff13c9445760 (R)
ffffff13e100f000 (R)
ffffff13c4d48be0 (R)
ffffff13c8e77b20 (R)
ffffff13d6246020 (R)
ffffff13df09cb60 (R)
ffffff13a9de64c0 (R)
ffffff13c9336760 (R)
ffffff13d490c160 (R)
ffffff13c4d023e0 (R)
ffffff13c0d5c0e0 (R)
ffffff13df0a9b20 (R)
ffffff13c4d02b20 (R)
ffffff13c8c0b500 (R)
ffffff13e26a5860 (R)
ffffff13c9312880 (R)
ffffff137d65f460 (R)
ffffff13d62e8b80 (R)
ffffff13d4910c20 (R)
ffffff13c9319100 (R)
ffffff13a9e2b760 (R)
ffffff13d47ed8c0 (R)
ffffff13dc671880 (R)
ffffff13c8e77040 (R)
ffffff13df0e04c0 (R)
ffffff13da6294c0 (R)
ffffff13d62f9780 (R)
ffffff13c9d8f440 (R)
ffffff13d4b2a7a0 (R)
ffffff13dc7a60e0 (R)
ffffff13c94473a0 (R)
ffffff13e5b2d500 (R)
ffffff13c577b3a0 (R)
ffffff13801d74a0 (R)
ffffff13df14c040 (R)
ffffff13c4d61160 (R)
ffffff13df1bbc20 (R)
ffffff13dd507020 (R)
ffffff13df1fc0e0 (R)
ffffff13d6248ae0 (R)
ffffff13df0b0ae0 (R)
ffffff13e569e840 (R)
ffffff13c4b66480 (R)
ffffff13925b4140 (R)
ffffff13dd533420 (R)
ffffff13db1c7440 (R)
ffffff13e69ac100 (R)
ffffff13dbcde760 (R)
ffffff13c4d56020 (R)
ffffff1391fa47c0 (R)
ffffff13e101ac00 (R)
ffffff13c4cfe7c0 (R)
ffffff13dd50f3a0 (R)
ffffff13d622c100 (R)
ffffff13d48754e0 (R)
ffffff13a9df0c40 (R)
ffffff13e687d4c0 (R)
ffffff13db1ccb60 (R)
ffffff13df212060 (R)
ffffff13c95c0440 (R)
ffffff13d61ff0c0 (R)
ffffff13d4b303c0 (R)
ffffff13c4d68140 (R)
ffffff1391f91b40 (R)
ffffff1364cee480 (R)
ffffff13d48b9b40 (R)
ffffff13dc713020 (R)
ffffff13db1edbc0 (R)
ffffff13e5b2c180 (R)
ffffff13df09b0c0 (R)
ffffff13e19e6400 (R)
ffffff13d61ff800 (R)
ffffff13dd53cb40 (R)
ffffff13c4b634c0 (R)
ffffff13c4b61c20 (R)
ffffff13c4cfbb80 (R)
ffffff13c58ca400 (R)
ffffff13c9315120 (R)
ffffff13d62117c0 (R)
ffffff1391f8ab80 (R)
ffffff13c9c11400 (R)
ffffff13e19b3040 (R)
ffffff13d62f37a0 (R)
ffffff13c574bb00 (R)
ffffff13df0ee480 (R)
ffffff13ab04d000 (R)
ffffff13dbb76120 (R)
ffffff13e19e4b60 (R)
ffffff13c58d9780 (R)
ffffff13df1df3a0 (R)
ffffff13c577b740 (R)
ffffff13df14c780 (R)
ffffff1391fa3b80 (R)
ffffff13a9db0100 (R)
ffffff13e101a4c0 (R)
ffffff13e68627e0 (R)
ffffff13c4d5a8c0 (R)
ffffff13c4d48840 (R)
ffffff13dd50f740 (R)
ffffff13df0e0860 (R)
ffffff13c0d2ac20 (R)
ffffff1408cdcb60 (R)
ffffff1408cce100 (R)
ffffff13e26c77e0 (R)
ffffff13d622c4a0 (R)
ffffff13ab040020 (R)
ffffff137d87d0e0 (R)
ffffff13d4910880 (R)
ffffff13d47cebe0 (R)
ffffff13c0d29160 (R)
ffffff13c9336020 (R)
ffffff13c8bd8b60 (R)
ffffff13d487bbc0 (R)
ffffff13d62010a0 (R)
ffffff137d65f800 (R)
ffffff13dc713b00 (R)
ffffff13e100f3a0 (R)
ffffff13df199ba0 (R)
ffffff13c4b61880 (R)
ffffff13c4b64be0 (R)
ffffff13aad443e0 (R)
ffffff13df1990c0 (R)
ffffff13801de7c0 (R)
ffffff137d87d820 (R)
ffffff13df00c520 (R)
ffffff13d47c7c20 (R)
ffffff13e68734e0 (R)
ffffff13c9547140 (R)
ffffff13d4853b00 (R)
ffffff13da671140 (R)
ffffff13e19a6b40 (R)
ffffff1391fa6400 (R)
ffffff13d484b400 (R)
ffffff13d47ce840 (R)
ffffff13c4b63120 (R)
ffffff13dc600400 (R)
ffffff13d6256c40 (R)
ffffff13c4b63860 (R)
ffffff13df2133e0 (R)
ffffff13c58f60a0 (R)
ffffff13c4b608a0 (R)
ffffff13c4b6a7e0 (R)
ffffff13c58e3820 (R)
ffffff13e1017c20 (R)
ffffff13df09bba0 (R)
ffffff13dc51fb20 (R)
ffffff137d69ebe0 (R)
ffffff13c9599b60 (R)
ffffff13d6246b00 (R)
ffffff13e26d4000 (R)
ffffff13df13eb80 (R)
ffffff13c0d5a120 (R)
ffffff13c4bf67e0 (R)
ffffff13e19eb020 (R)
ffffff13c4b64100 (R)
ffffff13924847c0 (R)
ffffff13c4bbb740 (R)
ffffff13dc7a6820 (R)
ffffff13dbb60500 (R)
ffffff13e0ffb760 (R)
ffffff13dc7133c0 (R)
ffffff13c4b6a0a0 (R)
ffffff13d95fa7e0 (R)
ffffff13c4b5f180 (R)
ffffff13d48c7c60 (R)
ffffff13e687d860 (R)
ffffff13d4888460 (R)
ffffff13c4b61140 (R)
ffffff13e6873c20 (R)
ffffff13df086880 (R)
ffffff13c4b69ba0 (R)
ffffff13c4b6a440 (R)
ffffff13a9de6860 (R)
ffffff13e68cc3e0 (R)
ffffff1391f8a7e0 (R)
ffffff13c4b66bc0 (R)
ffffff13c4b60500 (R)
ffffff13e4a5c160 (R)
ffffff13e4a60c20 (R)
ffffff13c47f8760 (R)
ffffff13e5b467e0 (R)
ffffff13df08d860 (R)
ffffff13d490c500 (R)
ffffff1408d0a120 (R)
ffffff13c8c0bc40 (R)
ffffff13c0d5bbe0 (R)
ffffff13c4cfb7e0 (R)
ffffff13db1f2460 (R)
ffffff13925a4be0 (R)
ffffff13c932fb20 (R)
ffffff13df17e740 (R)
ffffff13d4b3e4e0 (R)
ffffff13c9312c20 (R)
ffffff13df08d4c0 (R)
ffffff13e19b33e0 (R)
ffffff13c58e30e0 (R)
ffffff13c9329080 (R)
ffffff13c926cba0 (R)
ffffff13c9c117a0 (R)
ffffff13925044a0 (R)
ffffff13925ae000 (R)
ffffff13925a7ba0 (R)
ffffff13925acb20 (R)
ffffff137ff4d140 (R)
ffffff1364ce3be0 (R)
ffffff1391fa37e0 (R)
ffffff137d1a8ae0 (R)
ffffff134e0cc4c0 (R)
ffffff13925ac040 (R)
ffffff13925ad3c0 (R)
ffffff13c4ba1420 (R)
ffffff13da590840 (R)
ffffff13e19eb3c0 (R)
ffffff13e69b80c0 (R)
ffffff13e5b20760 (R)
ffffff13df1f2100 (R)
ffffff13df0a93e0 (R)
ffffff13d4910140 (R)
ffffff13925004c0 (R)
ffffff13c58e90c0 (R)
ffffff137519b0e0 (R)
ffffff13d4cbec60 (R)
ffffff1392504100 (R)
ffffff13df0e0120 (R)
ffffff13df0ea4a0 (R)
ffffff13df1a93c0 (R)
ffffff13da590be0 (W)
The following thread grabs reader lock(exported_lock) at http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/nfs/nfs_server.c#1668 in common_dispatch(), but checkexport() tries to grab the same reader lock again, thus blocked due to thread ffffff13da590be0 trying to grab a writer lock in exportfs().
ffffff13df08d860 SLEEP RWLOCK 1
swtch+0x141
turnstile_block+0x21a
rw_enter_sleep+0x236
checkexport+0x3b
rfs3_rename+0x117
common_dispatch+0x5a0
rfs_dispatch+0x2d
svc_getreq+0x1c1
svc_run+0xe0
svc_do_run+0x8e
nfssys+0x111
_sys_sysenter_post_swapgs+0x153
It seems this issue was introduced by https://www.illumos.org/issues/6090.
Related issues
Updated by 
Updated by