Project

General

Profile

Actions

Bug #5319

closed

smb_oplock_acquire thread deadlock

Added by Gordon Ross about 7 years ago. Updated about 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Start date:
2014-11-13
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

Bayard Bell added a comment - 05/Aug/13 6:18 PM - edited

This is my attempt to restate the problem, building on JB's extensive analysis. A fix analysis will follow this synthesis of root cause analysis.
Fundamentally, looking at smb_oplock_acquire(), it is clear that smb_range_check() is called with ol->ol_mutex held (usr/src/uts/common/fs/smbsrv/smb_oplock.c line 233) and node->vp->vnbllock held as a writer (usr/src/uts/common/fs/smbsrv/smb_oplock.c line 236, usr/src/uts/common/nblock.c line 48).
A large number of threads (1003) are blocking on ol->ol_mutex from smb_oplock_break(), while a further batch of thirteen threads are blocked on node->vp->vnbllock, ten from rfs3_read(), the remainder of which are:

> ffffff22dd4beb40::whatthread | ::stacks -C smb_oplock_break | ::stacks -cnbl_start_crit | ::stacks -C rfs3_read | ::findstack -v
stack pointer for thread ffffff00f7368c60: ffffff00f7368580
[ ffffff00f7368580 _resume_from_idle+0xf1() ]
  ffffff00f73685b0 swtch+0x145()
  ffffff00f7368660 turnstile_block+0x760(0, 0, ffffff22dd4beba0, fffffffffbc08ef0, 0, 0)
  ffffff00f73686d0 rw_enter_sleep+0x1a3(ffffff22dd4beba0, 0)
  ffffff00f73686f0 nbl_start_crit+0x19(ffffff22dd4beb40, 0)
  ffffff00f7368790 fs_frlock+0x229(ffffff22dd4beb40, 2a, ffffff00f7368960, 10002, 0, ffffff00f73688e0, ffffff35974803a0, ffffffffc03b6fb0)
  ffffff00f7368830 zfs_frlock+0xd9(ffffff22dd4beb40, 2a, ffffff00f7368960, 10002, 0, ffffff00f73688e0, ffffff35974803a0, ffffffffc03b6fb0)
  ffffff00f73688c0 fop_frlock+0x8c(ffffff22dd4beb40, 2a, ffffff00f7368960, 10002, 0, ffffff00f73688e0, ffffff35974803a0, ffffffffc03b6fb0)
  ffffff00f7368950 smb_vop_frlock+0x7d(ffffff22dd4beb40, ffffff35974803a0, 10002, ffffff00f7368960)
  ffffff00f73689e0 smb_fsop_frlock+0xdf(ffffff30583b0368, ffffff22683b7800, 0, ffffff35974803a0)
  ffffff00f7368a70 smb_lock_range+0x1b2(ffffff225550a348, 7fffffa7, 14, 0, 65)
  ffffff00f7368ae0 smb_com_locking_andx+0x318(ffffff225550a348)
  ffffff00f7368b80 smb_dispatch_request+0x4a6(ffffff225550a348)
  ffffff00f7368bb0 smb_session_worker+0x6c(ffffff225550a348)
  ffffff00f7368c40 taskq_d_thread+0xb1(ffffff369d311768)
  ffffff00f7368c50 thread_start+8()
stack pointer for thread ffffff2236856200: ffffff00f8b1f8c0
[ ffffff00f8b1f8c0 _resume_from_idle+0xf1() ]
  ffffff00f8b1f8f0 swtch+0x145()
  ffffff00f8b1f9a0 turnstile_block+0x760(ffffff225b25fa50, 0, ffffff22dd4beba0,  fffffffffbc08ef0, 0, 0)
  ffffff00f8b1fa10 rw_enter_sleep+0x1a3(ffffff22dd4beba0, 0)
  ffffff00f8b1fa30 nbl_start_crit+0x19(ffffff22dd4beb40, 0)
  ffffff00f8b1fa90 fs_shrlock+0xbe(ffffff22dd4beb40, 2b, ffffff00f8b1fc20, 2001, ffffff22c2e43b90, 0)
  ffffff00f8b1fb00 fop_shrlock+0x74(ffffff22dd4beb40, 2b, ffffff00f8b1fc20, 2001, ffffff22c2e43b90, 0)
  ffffff00f8b1fca0 vn_openat+0x590(8047ae7, 0, 2001, 0, ffffff00f8b1fce8, 0, 12, 0, 3)
  ffffff00f8b1fe00 copen+0x435(ffd19553, 8047ae7, 2001, 0)
  ffffff00f8b1fe30 open64+0x34(8047ae7, 0, 0)
  ffffff00f8b1fec0 dtrace_systrace_syscall32+0x11a(8047ae7, 0, 0, 1c3, 1, 0)
  ffffff00f8b1ff10 _sys_sysenter_post_swapgs+0x149()
stack pointer for thread ffffff21ed015e00: ffffff00f61bdaf0
[ ffffff00f61bdaf0 _resume_from_idle+0xf1() ]
  ffffff00f61bdb20 swtch+0x145()
  ffffff00f61bdbd0 turnstile_block+0x760(ffffff225b25fa50, 1, ffffff22dd4beba0,  fffffffffbc08ef0, 0, 0)
  ffffff00f61bdc40 rw_enter_sleep+0x205(ffffff22dd4beba0, 1)
  ffffff00f61bdc60 nbl_start_crit+0x19(ffffff22dd4beb40, 1)
  ffffff00f61bdde0 vn_renameat+0x292(0, 8047ae3, 0, 8047aec, 0)
  ffffff00f61bde10 vn_rename+0x2f(8047ae3, 8047aec, 0)
  ffffff00f61bde30 rename+0x17(8047ae3, 8047aec)
  ffffff00f61bdec0 dtrace_systrace_syscall32+0x11a(8047ae3, 8047aec, ffffffff, 1c3, 1, 0)
  ffffff00f61bdf10 _sys_sysenter_post_swapgs+0x149()

Actions

Also available in: Atom PDF