Project

General

Profile

Bug #9342

ZFS device remap stuck in dmu_tx_wait()

Added by Brad Lewis over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
2018-03-23
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

The problem is that the thread doing dmu_object_remap_one_indirect() is holding the dn_struct_rwlock, while calling dmu_tx_assign(). This causes a deadlock, because the dmu_tx_assign() may need to wait for a txg to pass (e.g. if this dnode is already assigned to a previous txg), but the other thread that has assigned this dnode to a previous txg is holding up that txg until it can get the dn_struct_rwlock.

The fix is to restructure the remap code to not hold the dn_struct_rwlock while assigning the tx.


> ffffff4c38fd04c0::findstack -v
stack pointer for thread ffffff4c38fd04c0: ffffff0179a9b930
[ ffffff0179a9b930 _resume_from_idle+0x112() ]
  ffffff0179a9b960 swtch+0x141()
  ffffff0179a9b9a0 cv_wait+0x70(ffffff877f9192d8, ffffff877f919218)
  ffffff0179a9b9f0 dmu_tx_wait+0x11b(ffffff8fb4d6a500)
  ffffff0179a9ba30 dmu_tx_assign+0x55(ffffff8fb4d6a500, 1)
  ffffff0179a9ba90 dmu_object_remap_one_indirect+0x11b(ffffff4c3478ed80, ffffff877f919140, b3774, 140000000)
  ffffff0179a9bb00 dmu_object_remap_indirects+0xc5(ffffff4c3478ed80, 74, b3774)
  ffffff0179a9bb50 dmu_objset_remap_indirects_impl+0x52(ffffff4c3478ed80, b3774)
  ffffff0179a9bba0 dmu_objset_remap_indirects+0x196(ffffff5806c41000)
  ffffff0179a9bbe0 zfs_ioc_remap+0x5a(ffffff5806c41000, ffffff4c9133cc38, ffffff4c91335778)
  ffffff0179a9bc80 zfsdev_ioctl+0x1b2(11d00000000, 5a47, 80466b8, 100003, ffffff82c16679d0, ffffff0179a9be68)
  ffffff0179a9bcc0 cdev_ioctl+0x39(11d00000000, 5a47, 80466b8, 100003, ffffff82c16679d0, ffffff0179a9be68)
  ffffff0179a9bd10 spec_ioctl+0x60(ffffff4c27ce1580, 5a47, 80466b8, 100003, ffffff82c16679d0, ffffff0179a9be68, 0)
  ffffff0179a9bda0 fop_ioctl+0x55(ffffff4c27ce1580, 5a47, 80466b8, 100003, ffffff82c16679d0, ffffff0179a9be68, 0)
  ffffff0179a9bec0 ioctl+0x9b(6, 5a47, 80466b8)
  ffffff0179a9bf10 _sys_sysenter_post_swapgs+0x149()

> ffffff8fb4d6a500::print -a dmu_tx_t tx_needassign_txh->txh_dnode
ffffffd195fa5530 tx_needassign_txh->txh_dnode = 0xffffff877f919140


this thread has the dn_struct_rwlock for writer, and
is waiting to get a tx on the dnode (dn_assigned_txg)


> ffffff4c28234860::findstack -v
stack pointer for thread ffffff4c28234860: ffffff0179e338c0
[ ffffff0179e338c0 _resume_from_idle+0x112() ]
  ffffff0179e338f0 swtch+0x141()
  ffffff0179e33990 turnstile_block+0x21a(0, 1, ffffff877f919140, fffffffffbc10460, 0, 0)
  ffffff0179e33a00 rw_enter_sleep+0x236(ffffff877f919140, 1)
  ffffff0179e33ab0 dmu_buf_hold_array_by_dnode+0x59(ffffff877f919140, 0, 2000, 0, fffffffff7adba70, ffffff0179e33afc, ffffff0179e33af0, 0)
  ffffff0179e33b50 dmu_write_uio_dnode+0x5f(ffffff877f919140, ffffff0179e33e60, 2000, ffffff8324132c80)
  ffffff0179e33ba0 dmu_write_uio_dbuf+0x5d(ffffff4c89236ec0, ffffff0179e33e60, 2000, ffffff8324132c80)
  ffffff0179e33db0 zfs_write+0xb9a(ffffff8bbadcf800, ffffff0179e33e60, 0, ffffff4c33498748, 0)
  ffffff0179e33e30 fop_write+0x5b(ffffff8bbadcf800, ffffff0179e33e60, 0, ffffff4c33498748, 0)
  ffffff0179e33f00 write+0x250(230, fffffd7fd2ff7ce0, 2000)
  ffffff0179e33f10 sys_syscall+0x177()

> ffffff8324132c80::walk list | ::print dmu_tx_hold_t txh_dnode
txh_dnode = 0xffffff877f919140
txh_dnode = 0xffffff877f919140

this thread has a tx on the dnode, and is trying to do:
rw_enter(&dn->dn_struct_rwlock, RW_READER);

Also available in: Atom PDF