Project

General

Profile

Bug #3951

ztest hang when running dedup test

Added by Christopher Siden over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Category:
zfs - Zettabyte File System
Start date:
2013-08-01
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

Analysis by George Wilson:

While running ztest I hit a deadman timer assertion. From the corefile we see
that we're trying to do some I/O:

> ::zio_state -r
ADDRESS         TYPE  STAGE            WAITER           TIME_ELAPSED
cd89910         NULL  CHECKSUM_VERIFY  17e              -            
 d28d600        READ  VDEV_IO_START    -                -            
  e8f1018       READ  VDEV_IO_START    -                -            
   e8d20e0      READ  VDEV_IO_START    -                -            
8813378         NULL  CHECKSUM_VERIFY  137              -            
 e8ee5e0        READ  VDEV_IO_START    -                -            
  e91e0e0       READ  VDEV_IO_START    -                -            
   e8fac08      READ  VDEV_IO_START    -                -            
d298c20         NULL  CHECKSUM_VERIFY  15e              -            
 e92ccd8        FREE  ISSUE_ASYNC      -                -            
 e914680        FREE  ISSUE_ASYNC      -                -            
 e90a648        FREE  ISSUE_ASYNC      -                -            
 d279128        FREE  ISSUE_ASYNC      -                -            
 e8f0bd0        FREE  ISSUE_ASYNC      -                -            
 e8fd348        FREE  ISSUE_ASYNC      -                -            
 ceab658        FREE  ISSUE_ASYNC      -                -            
 e930ce8        FREE  ISSUE_ASYNC      -                -            
 e90a358        FREE  ISSUE_ASYNC      -                -            
 e90f958        FREE  ISSUE_ASYNC      -                -            
 d277ce0        FREE  ISSUE_ASYNC      -                -            
 e90b940        FREE  ISSUE_ASYNC      -                -            
 e917988        FREE  ISSUE_ASYNC      -                -            
 d28d310        FREE  ISSUE_ASYNC      -                -            
 e8e8718        FREE  ISSUE_ASYNC      -                -            
 d277410        FREE  ISSUE_ASYNC      -                -            
 d130018        FREE  ISSUE_ASYNC      -                -            
 e92ece0        FREE  ISSUE_ASYNC      -                -            
 e8c9998        FREE  ISSUE_ASYNC      -                -            
 e90ac28        FREE  ISSUE_ASYNC      -                -            
 e90f378        FREE  ISSUE_ASYNC      -                -            
 e91c9a0        FREE  ISSUE_ASYNC      -                -            
 e8f7bf8        FREE  ISSUE_ASYNC      -                -            
 e8ef2f8        FREE  ISSUE_ASYNC      -                -            
 e914c60        FREE  ISSUE_ASYNC      -                -            
 d27ba08        FREE  ISSUE_ASYNC      -                -            
 e90e080        FREE  ISSUE_ASYNC      -                -            
 e8f5608        FREE  ISSUE_ASYNC      -                -            
 879c998        FREE  ISSUE_ASYNC      -                -            
 e90b070        FREE  ISSUE_ASYNC      -                -            
 e8e4700        FREE  ISSUE_ASYNC      -                -            
 e8f5028        FREE  ISSUE_ASYNC      -                -            
 e92e700        FREE  ISSUE_ASYNC      -                -            
 e8d1c98        FREE  ISSUE_ASYNC      -                -            
 d280d08        FREE  ISSUE_ASYNC      -                -            
 e90fc48        FREE  ISSUE_ASYNC      -                -            
 e916690        FREE  ISSUE_ASYNC      -                -            
 e911670        FREE  ISSUE_ASYNC      -                -            
 e8f28f0        FREE  ISSUE_ASYNC      -                -            
 e921cb0        FREE  ISSUE_ASYNC      -                -            
 e92a108        FREE  ISSUE_ASYNC      -                -            
 e8db108        FREE  ISSUE_ASYNC      -                -            
 e8fb050        FREE  ISSUE_ASYNC      -                -            
 d2c0c88        FREE  ISSUE_ASYNC      -                -            
 e914970        FREE  ISSUE_ASYNC      -                -            
 e916c70        FREE  ISSUE_ASYNC      -                -            
 cf72c28        FREE  ISSUE_ASYNC      -                -            
 e8ec730        FREE  ISSUE_ASYNC      -                -            
 e8d73e8        FREE  ISSUE_ASYNC      -                -            
 e915688        FREE  ISSUE_ASYNC      -                -            
 e8dccd0        FREE  ISSUE_ASYNC      -                -            
 d13fc00        FREE  ISSUE_ASYNC      -                -            
 e9180c0        FREE  ISSUE_ASYNC      -                -            
 d144c20        FREE  ISSUE_ASYNC      -                -            
 e9173a8        FREE  ISSUE_ASYNC      -                -            
 e923cb8        FREE  ISSUE_ASYNC      -                -            
 d277120        FREE  ISSUE_ASYNC      -                -            
 e8e7a00        FREE  ISSUE_ASYNC      -                -            
 d27e430        FREE  ISSUE_ASYNC      -                -            
 e8e36f8        FREE  ISSUE_ASYNC      -                -            
 e8d60f0        FREE  ISSUE_ASYNC      -                -            
 e8f8c00        FREE  ISSUE_ASYNC      -                -            
 e92e410        FREE  ISSUE_ASYNC      -                -            
 e914390        FREE  ISSUE_ASYNC      -                -            
 e9186a0        FREE  ISSUE_ASYNC      -                -            
 e9160b0        FREE  ISSUE_ASYNC      -                -            
 e91b0c8        FREE  ISSUE_ASYNC      -                -            
 e8dacc0        FREE  ISSUE_ASYNC      -                -            
 e90a068        FREE  ISSUE_ASYNC      -                -            
 e92a3f8        FREE  ISSUE_ASYNC      -                -            
 e8e9430        FREE  ISSUE_ASYNC      -                -            
 d28fbf0        FREE  ISSUE_ASYNC      -                -            
 d2700f8        FREE  ISSUE_ASYNC      -                -            
 e9213e0        FREE  ISSUE_ASYNC      -                -            
 e92e9f0        FREE  ISSUE_ASYNC      -                -            
 e8f58f8        FREE  ISSUE_ASYNC      -                -            
 e8e7420        FREE  ISSUE_ASYNC      -                -            
 e8f18e8        FREE  ISSUE_ASYNC      -                -            
 d1569f0        FREE  ISSUE_ASYNC      -                -            
 e912098        FREE  ISSUE_ASYNC      -                -            
 e92c6f8        FREE  ISSUE_ASYNC      -                -            
 e8f6bf0        FREE  ISSUE_ASYNC      -                -            
 e8f6900        FREE  ISSUE_ASYNC      -                -            
 e8f5318        FREE  ISSUE_ASYNC      -                -            
 d29a360        FREE  ISSUE_ASYNC      -                -            
 e8e9720        FREE  ISSUE_ASYNC      -                -            
 e8f8620        FREE  ISSUE_ASYNC      -                -            
 d1338f0        FREE  ISSUE_ASYNC      -                -            
 cdc9048        FREE  ISSUE_ASYNC      -                -            
 d2878d0        FREE  ISSUE_ASYNC      -                -            
 d14f6d8        FREE  ISSUE_ASYNC      -                -            
 d15e730        FREE  ISSUE_ASYNC      -                -            
 e912388        FREE  ISSUE_ASYNC      -                -            
 d29f088        FREE  ISSUE_ASYNC      -                -            
 d15b718        FREE  ISSUE_ASYNC      -                -            
 856a5e8        FREE  ISSUE_ASYNC      -                -            
 d27b428        FREE  ISSUE_ASYNC      -                -            
 d13bbe8        FREE  ISSUE_ASYNC      -                -            
 8813958        FREE  ISSUE_ASYNC      -                -            
 e8eb148        FREE  ISSUE_ASYNC      -                -            
 d276118        FREE  ISSUE_ASYNC      -                -            
 d298930        FREE  ISSUE_ASYNC      -                -            
 e91b3b8        FREE  ISSUE_ASYNC      -                -            
 e8e7130        FREE  ISSUE_ASYNC      -                -            
 e91f3d8        FREE  ISSUE_ASYNC      -                -            
 e92b110        FREE  ISSUE_ASYNC      -                -            
 e8fdc18        FREE  ISSUE_ASYNC      -                -            
 e8e3118        FREE  ISSUE_ASYNC      -                -            
 e90f668        FREE  ISSUE_ASYNC      -                -            
 a1d3688        FREE  ISSUE_ASYNC      -                -            
 e8d2ca0        FREE  ISSUE_ASYNC      -                -            
 cb835e8        FREE  ISSUE_ASYNC      -                -            
 e9216d0        FREE  ISSUE_ASYNC      -                -            
 d15c140        FREE  ISSUE_ASYNC      -                -            
 e924cc0        FREE  ISSUE_ASYNC      -                -            
 e918c80        FREE  ISSUE_ASYNC      -                -            
 e90e370        FREE  ISSUE_ASYNC      -                -            
 d142050        FREE  ISSUE_ASYNC      -                -            
 e92b400        FREE  ISSUE_ASYNC      -                -            
 e909c20        FREE  ISSUE_ASYNC      -                -            
 d2746f0        FREE  ISSUE_ASYNC      -                -            
 e915398        FREE  ISSUE_ASYNC      -                -            
 e9236d8        FREE  ISSUE_ASYNC      -                -            
 d29b948        FREE  ISSUE_ASYNC      -                -            
 e916980        FREE  ISSUE_ASYNC      -                -            
 e8e9a10        FREE  ISSUE_ASYNC      -                -            
 e930708        FREE  ISSUE_ASYNC      -                -            
 d1539d8        FREE  ISSUE_ASYNC      -                -            
 e8e39e8        FREE  ISSUE_ASYNC      -                -            
 e90c948        FREE  ISSUE_ASYNC      -                -            
 e909640        FREE  ISSUE_ASYNC      -                -            
 e8fa338        FREE  ISSUE_ASYNC      -                -            
 e90bc30        FREE  ISSUE_ASYNC      -                -            
 d13c320        FREE  ISSUE_ASYNC      -                -            
 e8f7908        FREE  ISSUE_ASYNC      -                -            
 d1589f8        FREE  ISSUE_ASYNC      -                -            
 cff3970        FREE  ISSUE_ASYNC      -                -            
 d15b428        FREE  ISSUE_ASYNC      -                -            
 e911380        FREE  ISSUE_ASYNC      -                -            
 e9219c0        FREE  ISSUE_ASYNC      -                -            
 e8d13c8        FREE  ISSUE_ASYNC      -                -            
 e8fd058        FREE  ISSUE_ASYNC      -                -            
 d14ecb0        FREE  ISSUE_ASYNC      -                -            
 e9210f0        FREE  ISSUE_ASYNC      -                -            
 e924100        FREE  ISSUE_ASYNC      -                -            
 e92c408        FREE  ISSUE_ASYNC      -                -            
 d29e660        FREE  ISSUE_ASYNC      -                -            
 cdc9918        FREE  ISSUE_ASYNC      -                -            
 e91b6a8        FREE  ISSUE_ASYNC      -                -            
 e92bcd0        FREE  ISSUE_ASYNC      -                -            
 d129000        FREE  ISSUE_ASYNC      -                -            
 e911c50        FREE  ISSUE_ASYNC      -                -            
 e91e3d0        FREE  ISSUE_ASYNC      -                -            
 d15dd08        FREE  ISSUE_ASYNC      -                -            
 e8ef5e8        FREE  ISSUE_ASYNC      -                -            
 e8da6e0        FREE  ISSUE_ASYNC      -                -            
 d3df360        FREE  ISSUE_ASYNC      -                -            
 e92a9d8        FREE  ISSUE_ASYNC      -                -            
 e8d70f8        FREE  ISSUE_ASYNC      -                -            
 e91d3c8        FREE  ISSUE_ASYNC      -                -            
 e8d50e8        FREE  ISSUE_ASYNC      -                -            
 d2d09b8        FREE  ISSUE_ASYNC      -                -            
 e8e5128        FREE  ISSUE_ASYNC      -                -            
 e8d29b0        FREE  ISSUE_ASYNC      -                -            
 e8d79c8        FREE  DDT_FREE         -                -            
 d2a20a0        FREE  ISSUE_ASYNC      -                -            
 e91f6c8        FREE  ISSUE_ASYNC      -                -            
 e91dc98        FREE  ISSUE_ASYNC      -                -            
 e9243f0        FREE  ISSUE_ASYNC      -                -            
 d146c28        FREE  ISSUE_ASYNC      -                -            
 e8ee8d0        FREE  ISSUE_ASYNC      -                -            
 e90c078        FREE  DDT_FREE         -                -            
 e915c68        FREE  ISSUE_ASYNC      -                -            
 e90e660        FREE  DDT_FREE         -                -            
 e8f0300        FREE  DDT_FREE         -                -            
 e911960        FREE  ISSUE_ASYNC      -                -            
 e9233e8        FREE  ISSUE_ASYNC      -                -            
 e8e4410        FREE  DDT_FREE         -                -            
 e9230f8        FREE  DDT_FREE         -                -            
 d27e720        FREE  DDT_FREE         -                -            
 e915978        FREE  ISSUE_ASYNC      -                -            
 e8fa628        FREE  DDT_FREE         -                -            
 e90ec40        FREE  ISSUE_ASYNC      -                -            
 d155118        FREE  DDT_FREE         -                -            
 e92b9e0        FREE  DDT_FREE         -                -            
 e8ebd08        FREE  DDT_FREE         -                -            
 d1295e0        FREE  ISSUE_ASYNC      -                -            
 bd2c6b8        FREE  ISSUE_ASYNC      -                -            
 e91d0d8        FREE  DDT_FREE         -                -            
 e8eb438        FREE  DDT_FREE         -                -            
 e9239c8        FREE  DDT_FREE         -                -            
 e8f5be8        FREE  ISSUE_ASYNC      -                -            
 e8d63e0        FREE  DDT_FREE         -                -            
 cff3c60        FREE  DDT_FREE         -                -            
 d295340        FREE  ISSUE_ASYNC      -                -            
 d12a5e8        FREE  DDT_FREE         -                -            
 e91f0e8        FREE  DDT_FREE         -                -            
 d29b368        FREE  ISSUE_ASYNC      -                -            
 d15c720        FREE  ISSUE_ASYNC      -                -            
 e91c0d0        FREE  DDT_FREE         -                -            
 e8e3408        FREE  DDT_FREE         -                -            
 e917698        FREE  DDT_FREE         -                -            
 e8dc6f0        FREE  ISSUE_ASYNC      -                -            
 e92e120        FREE  DDT_FREE         -                -            
 e8fd928        FREE  DDT_FREE         -                -            
 d26eca8        FREE  DDT_FREE         -                -            
 e937a00        FREE  DDT_FREE         -                -            
 e8d76d8        FREE  DDT_FREE         -                -            
 e9309f8        FREE  DDT_FREE         -                -            
 e8e5ce8        FREE  DDT_FREE         -                -            
 e91b998        FREE  DDT_FREE         -                -            
 e8da100        FREE  DDT_FREE         -                -            
 d14e9c0        FREE  DDT_FREE         -                -            
 d12a008        FREE  ISSUE_ASYNC      -                -            
 e91d9a8        FREE  DDT_FREE         -                -            
 e8da9d0        FREE  DDT_FREE         -                -            
 e8d6cb0        FREE  DDT_FREE         -                -            
 d276408        FREE  DDT_FREE         -                -            
 86b66a0        FREE  DDT_FREE         -                -            
 e8f15f8        FREE  DDT_FREE         -                -            
 e9246e0        FREE  DDT_FREE         -                -            
 d274400        FREE  DDT_FREE         -                -            
 e909060        FREE  DDT_FREE         -                -            
 d26d6c0        FREE  DDT_FREE         -                -            
 e91bc88        FREE  DDT_FREE         -                -            
 e90cc38        FREE  DDT_FREE         -                -            
 e8f8040        FREE  DDT_FREE         -                -            
 e8fb630        FREE  DDT_FREE         -                -            
 e8d09a0        FREE  DDT_FREE         -                -            
 d292910        FREE  DDT_FREE         -                -            
 e8d7cb8        FREE  DDT_FREE         -                -            
 e8dc400        FREE  DDT_FREE         -                -            
 e91f9b8        FREE  DDT_FREE         -                -            
 e8f8910        FREE  DDT_FREE         -                -            
 d15e440        FREE  DDT_FREE         -                -            
 d2706d8        FREE  DDT_FREE         -                -            
 cde2950        FREE  DDT_FREE         -                -            
 e8e8138        FREE  DDT_FREE         -                -            
 e8f0010        FREE  DDT_FREE         -                -            
 e8f2310        FREE  DDT_FREE         -                -            
 e8d53d8        FREE  DDT_FREE         -                -            
 e8fa918        FREE  DDT_FREE         -                -            
 e91fca8        FREE  DDT_FREE         -                -            
 e8f6320        FREE  DDT_FREE         -                -            
 e8efbc8        FREE  DDT_FREE         -                -            
 d14e6d0        FREE  DDT_FREE         -                -            
 e90b360        FREE  DDT_FREE         -                -            
 e8f7618        FREE  DDT_FREE         -                -            
 e937cf0        FREE  DDT_FREE         -                -            
 e8eebc0        FREE  DDT_FREE         -                -            
 d28b8e0        FREE  DDT_FREE         -                -            
 e90f088        FREE  DDT_FREE         -                -            
 d29a070        FREE  DDT_FREE         -                -            
 d294918        FREE  DDT_FREE         -                -            
 e8e4120        FREE  DDT_FREE         -                -            
 e90a938        FREE  DDT_FREE         -                -            
 e8e49f0        FREE  DDT_FREE         -                -            
 d15d438        FREE  DDT_FREE         -                -            
 d13c610        FREE  DDT_FREE         -                -            
 8791960        FREE  DDT_FREE         -                -            
 e8f7038        FREE  DDT_FREE         -                -            
 d2719d0        FREE  DDT_FREE         -                -            
 e8f1bd8        FREE  DDT_FREE         -                -            
 d295630        FREE  DDT_FREE         -                -            
 e8e59f8        FREE  DDT_FREE         -                -            
 e92c118        FREE  DDT_FREE         -                -            
 d3e1658        FREE  DDT_FREE         -                -            
 e8f2020        FREE  DDT_FREE         -                -            
 e8e5708        FREE  DDT_FREE         -                -            
 e8e7710        FREE  DDT_FREE         -                -            
 e8eba18        FREE  DDT_FREE         -                -            
 e911090        FREE  DDT_FREE         -                -            
 894c340        FREE  DDT_FREE         -                -            
 e8e3cd8        FREE  DDT_FREE         -                -            
 d2cd9a0        FREE  DDT_FREE         -                -            
 d13c030        FREE  DDT_FREE         -                -            
 e8f2600        FREE  DDT_FREE         -                -            
 e8fb920        FREE  DDT_FREE         -                -            
 d295050        FREE  DDT_FREE         -                -            
8746960         NULL  OPEN             -                -            
856abc8         NULL  OPEN             -                -            
e8fd638         NULL  OPEN             -                -

From the corefile we see this:

> ::stacks -c zio_execute
THREAD   STATE    SOBJ        COUNT
f0       PARKED   CV             95
         libc.so.1`cond_wait_queue+0x60
         libc.so.1`__cond_wait+0x86
         libc.so.1`cond_wait+0x24
         libzpool.so.1`cv_wait+0x40
         libzpool.so.1`ddt_lookup+0xc6
         libzpool.so.1`zio_ddt_free+0x5c
         libzpool.so.1`zio_execute+0x13b
         libzpool.so.1`taskq_thread+0x9c
         libc.so.1`_thrp_setup+0x9b
         libc.so.1`_lwp_start

137      PARKED   CV              1
         libc.so.1`cond_wait_queue+0x60
         libc.so.1`__cond_wait+0x86
         libc.so.1`cond_wait+0x24
         libzpool.so.1`cv_wait+0x40
         libzpool.so.1`zio_wait+0x7f
         libzpool.so.1`dbuf_read+0x27f
         libzpool.so.1`dmu_buf_hold+0xae
         libzpool.so.1`zap_get_leaf_byblk+0x87
         libzpool.so.1`zap_deref_leaf+0xa4
         libzpool.so.1`fzap_length+0x31
         libzpool.so.1`zap_length_uint64+0x62
         libzpool.so.1`ddt_zap_lookup+0x3e
         libzpool.so.1`ddt_object_lookup+0x5a
         libzpool.so.1`ddt_lookup+0x14b
         libzpool.so.1`zio_ddt_free+0x5c
         libzpool.so.1`zio_execute+0x13b
         libzpool.so.1`taskq_thread+0x9c
         libc.so.1`_thrp_setup+0x9b
         libc.so.1`_lwp_start

We have a bunch of threads that are waiting for the ddt_load to finish to
complete and another set that are trying to load the ddt. Unfortunately we're
out of taskqs to do the work. This is because in vdev_file_io_start() we use
the ZIO_TYPE_FREE issue taskq to do asynchronous I/O but it can't complete the
I/O because all the FREE issue threads are already consumed.

Switching vdev_file_io_start() to use the system_taskq instead.

History

#1

Updated by Christopher Siden over 5 years ago

  • Status changed from In Progress to Closed
commit 2c1e2b4
Author: George Wilson <george.wilson@delphix.com>
Date:   Wed Aug 7 11:24:34 2013

    3949 ztest fault injection should avoid resilvering devices
    3950 ztest: deadman fires when we're doing a scan
    3951 ztest hang when running dedup test
    3952 ztest: ztest_reguid test and ztest_fault_inject don't place nice together
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>

Also available in: Atom PDF