Project

General

Profile

Bug #5823

ZFS: ZIO deadlock

Added by Simon Klinkert over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
zfs - Zettabyte File System
Start date:
2015-04-10
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

We’ve hit another ZFS deadlock involving at least the following two ZIO threads:

> ffffff001ea0fc40::findstack -v
stack pointer for thread ffffff001ea0fc40: ffffff001ea0f2b0
[ ffffff001ea0f2b0 _resume_from_idle+0xf1() ]
  ffffff001ea0f2e0 swtch+0x133()
  ffffff001ea0f380 turnstile_block+0x23b(0, 0, ffffff2400afd230, fffffffffbc07d60, 0, 0)
  ffffff001ea0f3f0 mutex_vector_enter+0x260(ffffff2400afd230)
  ffffff001ea0f460 dsl_dataset_block_kill+0x2d7()
  ffffff001ea0f4c0 dbuf_write_done+0x17e(ffffff053b6debb0, ffffff04e96ec3a0, ffffff507be34d90)
  ffffff001ea0f510 arc_write_done+0x10d(ffffff053b6debb0)
  ffffff001ea0f590 zio_done+0x4c0()
  ffffff001ea0f5d0 zio_execute+0x8b(ffffff053b6debb0)
  ffffff001ea0f630 zio_notify_parent+0xe0(ffffff053b6debb0, ffffff6caae3f420, 1)
  ffffff001ea0f6b0 zio_done+0x39b()
  ffffff001ea0f6f0 zio_execute+0x8b(ffffff6caae3f420)
  ffffff001ea0f750 zio_notify_parent+0xe0(ffffff6caae3f420, ffffff506d632370, 1)
  ffffff001ea0f7d0 zio_done+0x39b()
  ffffff001ea0f810 zio_execute+0x8b(ffffff506d632370)
  ffffff001ea0f870 zio_notify_parent+0xe0(ffffff506d632370, ffffff50570c5b68, 1)
  ffffff001ea0f8f0 zio_done+0x39b()
  ffffff001ea0f930 zio_execute+0x8b(ffffff50570c5b68)
  ffffff001ea0f9d0 vdev_queue_io_to_issue+0x2a5()
  ffffff001ea0fa20 vdev_queue_io_done+0xa3(ffffff40590a0478)
  ffffff001ea0fa60 zio_vdev_io_done+0xc8(ffffff40590a0478)
  ffffff001ea0faa0 zio_execute+0x8b(ffffff40590a0478)
  ffffff001ea0fb30 taskq_thread+0x2d9(ffffff04f7a45d30)
  ffffff001ea0fb40 thread_start+8()

> ffffff04eb039800::findstack -v
stack pointer for thread ffffff04eb039800: ffffff0021e35700
[ ffffff0021e35700 _resume_from_idle+0xf1() ]
  ffffff0021e35730 swtch+0x133()
  ffffff0021e35760 cv_wait+0x6d(ffffff05293adfa0, ffffff05293adf98)
  ffffff0021e357a0 zio_wait+0x5e(ffffff05293adc98)
  ffffff0021e35810 dbuf_read+0x208()
  ffffff0021e35880 dmu_buf_hold+0xcf()
  ffffff0021e35910 zap_lockdir+0x6b()
  ffffff0021e35990 zap_lookup_norm+0x5e(ffffff04f7d9d340, 85964, ffffff6cb1bbdc90, 8, 1, ffffff0021e35a00, ffffff0000000000, 0, 0, 0)
  ffffff0021e359f0 zap_lookup+0x46(ffffff04f7d9d340, 85964, ffffff6cb1bbdc90, 8, 1, ffffff0021e35a00)
  ffffff0021e35a50 dsl_dataset_release_might_destroy+0x80()
  ffffff0021e35ab0 dsl_dataset_user_release_one+0xdc()
  ffffff0021e35b40 dsl_dataset_user_release+0x140(ffffff5a3dc19008, ffffff5a3dc19032, ffffff6cb1bbdc90, 0)
  ffffff0021e35bc0 dsl_dataset_user_release_tmp+0xd4(ffffff04e94e80c0, 8591f, ffffff6cb1bbdc90, 1)
  ffffff0021e35bf0 dsl_dataset_user_release_onexit+0x28()
  ffffff0021e35c30 zfs_onexit_destroy+0x41(ffffff5067349c20)
  ffffff0021e35c60 zfs_ctldev_destroy+0x1b(ffffff5067349c20, 8b)
  ffffff0021e35cc0 zfsdev_close+0x87(5f0000008b, 2403, 2, ffffff0f2567e800)
  ffffff0021e35cf0 dev_close+0x31()
  ffffff0021e35d30 device_close+0xa2(ffffff04fcd6f900, 2403, ffffff0f2567e800)
  ffffff0021e35dc0 spec_close+0x163(ffffff04fcd6f900, 2403, 1, 0, ffffff0f2567e800, 0)
  ffffff0021e35e30 fop_close+0x74()
  ffffff0021e35e70 closef+0x63(fffffff743533cd0)
  ffffff0021e35ee0 closeandsetf+0x325(d, 0)
  ffffff0021e35f00 close+0x13()
  ffffff0021e35f10 sys_syscall+0x17a()

ffffff001ea0fc40 blocks on a mutex which is held by ffffff04eb039800:

> ffffff2400afd230::mutex
            ADDR  TYPE             HELD MINSPL OLDSPL WAITERS
ffffff2400afd230 adapt ffffff04eb039800      -      -     yes

The mutex we are waiting for in dsl_dataset_block_kill() is:

mutex_enter(&ds->ds_prev->ds_lock);

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/dsl_dataset.c#199

ffffff04eb039800 is waiting for io_cv in zio_wait(). Someone else has to notify this parent ZIO. Thread ffffff04eb039800 is both, io_executor and io_waiter for ZIO ffffff05293adc98:

> ffffff05293adc98::zio -rcp
ADDRESS                                  TYPE  STAGE            WAITER          
ffffff05293adc98                         NULL  CHECKSUM_VERIFY  ffffff04eb039800
 ffffff5058852768                        READ  VDEV_IO_START    -
  ffffff11655e4b68                       READ  VDEV_IO_START    -
   ffffff40590a0478                      READ  VDEV_IO_DONE     -
> ffffff05293adc98::print -t zio_t io_executor
void *io_executor = 0xffffff04eb039800

The executor of the last ZIO is ffffff001ea0fc40:

> ffffff40590a0478::print -t zio_t io_executor
void *io_executor = 0xffffff001ea0fc40

Conclusion: I think the first thread should notify the second thread about completion of child I/O but dsl_dataset_block_kill() waits for a lock which is held by the other thread.

Unfortunately, I cannot share the whole crashdump, but I’ll see what I can do if there’re further questions. Please feel free to ask. I’m not running the latest Illumos but I think the code in this area hasn’t changed much.

Also available in: Atom PDF