Actions
Bug #3951
closed
ztest hang when running dedup test
Start date:
2013-08-01
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:
External Bug:
Description
Analysis by George Wilson:
While running ztest I hit a deadman timer assertion. From the corefile we see that we're trying to do some I/O: > ::zio_state -r ADDRESS TYPE STAGE WAITER TIME_ELAPSED cd89910 NULL CHECKSUM_VERIFY 17e - d28d600 READ VDEV_IO_START - - e8f1018 READ VDEV_IO_START - - e8d20e0 READ VDEV_IO_START - - 8813378 NULL CHECKSUM_VERIFY 137 - e8ee5e0 READ VDEV_IO_START - - e91e0e0 READ VDEV_IO_START - - e8fac08 READ VDEV_IO_START - - d298c20 NULL CHECKSUM_VERIFY 15e - e92ccd8 FREE ISSUE_ASYNC - - e914680 FREE ISSUE_ASYNC - - e90a648 FREE ISSUE_ASYNC - - d279128 FREE ISSUE_ASYNC - - e8f0bd0 FREE ISSUE_ASYNC - - e8fd348 FREE ISSUE_ASYNC - - ceab658 FREE ISSUE_ASYNC - - e930ce8 FREE ISSUE_ASYNC - - e90a358 FREE ISSUE_ASYNC - - e90f958 FREE ISSUE_ASYNC - - d277ce0 FREE ISSUE_ASYNC - - e90b940 FREE ISSUE_ASYNC - - e917988 FREE ISSUE_ASYNC - - d28d310 FREE ISSUE_ASYNC - - e8e8718 FREE ISSUE_ASYNC - - d277410 FREE ISSUE_ASYNC - - d130018 FREE ISSUE_ASYNC - - e92ece0 FREE ISSUE_ASYNC - - e8c9998 FREE ISSUE_ASYNC - - e90ac28 FREE ISSUE_ASYNC - - e90f378 FREE ISSUE_ASYNC - - e91c9a0 FREE ISSUE_ASYNC - - e8f7bf8 FREE ISSUE_ASYNC - - e8ef2f8 FREE ISSUE_ASYNC - - e914c60 FREE ISSUE_ASYNC - - d27ba08 FREE ISSUE_ASYNC - - e90e080 FREE ISSUE_ASYNC - - e8f5608 FREE ISSUE_ASYNC - - 879c998 FREE ISSUE_ASYNC - - e90b070 FREE ISSUE_ASYNC - - e8e4700 FREE ISSUE_ASYNC - - e8f5028 FREE ISSUE_ASYNC - - e92e700 FREE ISSUE_ASYNC - - e8d1c98 FREE ISSUE_ASYNC - - d280d08 FREE ISSUE_ASYNC - - e90fc48 FREE ISSUE_ASYNC - - e916690 FREE ISSUE_ASYNC - - e911670 FREE ISSUE_ASYNC - - e8f28f0 FREE ISSUE_ASYNC - - e921cb0 FREE ISSUE_ASYNC - - e92a108 FREE ISSUE_ASYNC - - e8db108 FREE ISSUE_ASYNC - - e8fb050 FREE ISSUE_ASYNC - - d2c0c88 FREE ISSUE_ASYNC - - e914970 FREE ISSUE_ASYNC - - e916c70 FREE ISSUE_ASYNC - - cf72c28 FREE ISSUE_ASYNC - - e8ec730 FREE ISSUE_ASYNC - - e8d73e8 FREE ISSUE_ASYNC - - e915688 FREE ISSUE_ASYNC - - e8dccd0 FREE ISSUE_ASYNC - - d13fc00 FREE ISSUE_ASYNC - - e9180c0 FREE ISSUE_ASYNC - - d144c20 FREE ISSUE_ASYNC - - e9173a8 FREE ISSUE_ASYNC - - e923cb8 FREE ISSUE_ASYNC - - d277120 FREE ISSUE_ASYNC - - e8e7a00 FREE ISSUE_ASYNC - - d27e430 FREE ISSUE_ASYNC - - e8e36f8 FREE ISSUE_ASYNC - - e8d60f0 FREE ISSUE_ASYNC - - e8f8c00 FREE ISSUE_ASYNC - - e92e410 FREE ISSUE_ASYNC - - e914390 FREE ISSUE_ASYNC - - e9186a0 FREE ISSUE_ASYNC - - e9160b0 FREE ISSUE_ASYNC - - e91b0c8 FREE ISSUE_ASYNC - - e8dacc0 FREE ISSUE_ASYNC - - e90a068 FREE ISSUE_ASYNC - - e92a3f8 FREE ISSUE_ASYNC - - e8e9430 FREE ISSUE_ASYNC - - d28fbf0 FREE ISSUE_ASYNC - - d2700f8 FREE ISSUE_ASYNC - - e9213e0 FREE ISSUE_ASYNC - - e92e9f0 FREE ISSUE_ASYNC - - e8f58f8 FREE ISSUE_ASYNC - - e8e7420 FREE ISSUE_ASYNC - - e8f18e8 FREE ISSUE_ASYNC - - d1569f0 FREE ISSUE_ASYNC - - e912098 FREE ISSUE_ASYNC - - e92c6f8 FREE ISSUE_ASYNC - - e8f6bf0 FREE ISSUE_ASYNC - - e8f6900 FREE ISSUE_ASYNC - - e8f5318 FREE ISSUE_ASYNC - - d29a360 FREE ISSUE_ASYNC - - e8e9720 FREE ISSUE_ASYNC - - e8f8620 FREE ISSUE_ASYNC - - d1338f0 FREE ISSUE_ASYNC - - cdc9048 FREE ISSUE_ASYNC - - d2878d0 FREE ISSUE_ASYNC - - d14f6d8 FREE ISSUE_ASYNC - - d15e730 FREE ISSUE_ASYNC - - e912388 FREE ISSUE_ASYNC - - d29f088 FREE ISSUE_ASYNC - - d15b718 FREE ISSUE_ASYNC - - 856a5e8 FREE ISSUE_ASYNC - - d27b428 FREE ISSUE_ASYNC - - d13bbe8 FREE ISSUE_ASYNC - - 8813958 FREE ISSUE_ASYNC - - e8eb148 FREE ISSUE_ASYNC - - d276118 FREE ISSUE_ASYNC - - d298930 FREE ISSUE_ASYNC - - e91b3b8 FREE ISSUE_ASYNC - - e8e7130 FREE ISSUE_ASYNC - - e91f3d8 FREE ISSUE_ASYNC - - e92b110 FREE ISSUE_ASYNC - - e8fdc18 FREE ISSUE_ASYNC - - e8e3118 FREE ISSUE_ASYNC - - e90f668 FREE ISSUE_ASYNC - - a1d3688 FREE ISSUE_ASYNC - - e8d2ca0 FREE ISSUE_ASYNC - - cb835e8 FREE ISSUE_ASYNC - - e9216d0 FREE ISSUE_ASYNC - - d15c140 FREE ISSUE_ASYNC - - e924cc0 FREE ISSUE_ASYNC - - e918c80 FREE ISSUE_ASYNC - - e90e370 FREE ISSUE_ASYNC - - d142050 FREE ISSUE_ASYNC - - e92b400 FREE ISSUE_ASYNC - - e909c20 FREE ISSUE_ASYNC - - d2746f0 FREE ISSUE_ASYNC - - e915398 FREE ISSUE_ASYNC - - e9236d8 FREE ISSUE_ASYNC - - d29b948 FREE ISSUE_ASYNC - - e916980 FREE ISSUE_ASYNC - - e8e9a10 FREE ISSUE_ASYNC - - e930708 FREE ISSUE_ASYNC - - d1539d8 FREE ISSUE_ASYNC - - e8e39e8 FREE ISSUE_ASYNC - - e90c948 FREE ISSUE_ASYNC - - e909640 FREE ISSUE_ASYNC - - e8fa338 FREE ISSUE_ASYNC - - e90bc30 FREE ISSUE_ASYNC - - d13c320 FREE ISSUE_ASYNC - - e8f7908 FREE ISSUE_ASYNC - - d1589f8 FREE ISSUE_ASYNC - - cff3970 FREE ISSUE_ASYNC - - d15b428 FREE ISSUE_ASYNC - - e911380 FREE ISSUE_ASYNC - - e9219c0 FREE ISSUE_ASYNC - - e8d13c8 FREE ISSUE_ASYNC - - e8fd058 FREE ISSUE_ASYNC - - d14ecb0 FREE ISSUE_ASYNC - - e9210f0 FREE ISSUE_ASYNC - - e924100 FREE ISSUE_ASYNC - - e92c408 FREE ISSUE_ASYNC - - d29e660 FREE ISSUE_ASYNC - - cdc9918 FREE ISSUE_ASYNC - - e91b6a8 FREE ISSUE_ASYNC - - e92bcd0 FREE ISSUE_ASYNC - - d129000 FREE ISSUE_ASYNC - - e911c50 FREE ISSUE_ASYNC - - e91e3d0 FREE ISSUE_ASYNC - - d15dd08 FREE ISSUE_ASYNC - - e8ef5e8 FREE ISSUE_ASYNC - - e8da6e0 FREE ISSUE_ASYNC - - d3df360 FREE ISSUE_ASYNC - - e92a9d8 FREE ISSUE_ASYNC - - e8d70f8 FREE ISSUE_ASYNC - - e91d3c8 FREE ISSUE_ASYNC - - e8d50e8 FREE ISSUE_ASYNC - - d2d09b8 FREE ISSUE_ASYNC - - e8e5128 FREE ISSUE_ASYNC - - e8d29b0 FREE ISSUE_ASYNC - - e8d79c8 FREE DDT_FREE - - d2a20a0 FREE ISSUE_ASYNC - - e91f6c8 FREE ISSUE_ASYNC - - e91dc98 FREE ISSUE_ASYNC - - e9243f0 FREE ISSUE_ASYNC - - d146c28 FREE ISSUE_ASYNC - - e8ee8d0 FREE ISSUE_ASYNC - - e90c078 FREE DDT_FREE - - e915c68 FREE ISSUE_ASYNC - - e90e660 FREE DDT_FREE - - e8f0300 FREE DDT_FREE - - e911960 FREE ISSUE_ASYNC - - e9233e8 FREE ISSUE_ASYNC - - e8e4410 FREE DDT_FREE - - e9230f8 FREE DDT_FREE - - d27e720 FREE DDT_FREE - - e915978 FREE ISSUE_ASYNC - - e8fa628 FREE DDT_FREE - - e90ec40 FREE ISSUE_ASYNC - - d155118 FREE DDT_FREE - - e92b9e0 FREE DDT_FREE - - e8ebd08 FREE DDT_FREE - - d1295e0 FREE ISSUE_ASYNC - - bd2c6b8 FREE ISSUE_ASYNC - - e91d0d8 FREE DDT_FREE - - e8eb438 FREE DDT_FREE - - e9239c8 FREE DDT_FREE - - e8f5be8 FREE ISSUE_ASYNC - - e8d63e0 FREE DDT_FREE - - cff3c60 FREE DDT_FREE - - d295340 FREE ISSUE_ASYNC - - d12a5e8 FREE DDT_FREE - - e91f0e8 FREE DDT_FREE - - d29b368 FREE ISSUE_ASYNC - - d15c720 FREE ISSUE_ASYNC - - e91c0d0 FREE DDT_FREE - - e8e3408 FREE DDT_FREE - - e917698 FREE DDT_FREE - - e8dc6f0 FREE ISSUE_ASYNC - - e92e120 FREE DDT_FREE - - e8fd928 FREE DDT_FREE - - d26eca8 FREE DDT_FREE - - e937a00 FREE DDT_FREE - - e8d76d8 FREE DDT_FREE - - e9309f8 FREE DDT_FREE - - e8e5ce8 FREE DDT_FREE - - e91b998 FREE DDT_FREE - - e8da100 FREE DDT_FREE - - d14e9c0 FREE DDT_FREE - - d12a008 FREE ISSUE_ASYNC - - e91d9a8 FREE DDT_FREE - - e8da9d0 FREE DDT_FREE - - e8d6cb0 FREE DDT_FREE - - d276408 FREE DDT_FREE - - 86b66a0 FREE DDT_FREE - - e8f15f8 FREE DDT_FREE - - e9246e0 FREE DDT_FREE - - d274400 FREE DDT_FREE - - e909060 FREE DDT_FREE - - d26d6c0 FREE DDT_FREE - - e91bc88 FREE DDT_FREE - - e90cc38 FREE DDT_FREE - - e8f8040 FREE DDT_FREE - - e8fb630 FREE DDT_FREE - - e8d09a0 FREE DDT_FREE - - d292910 FREE DDT_FREE - - e8d7cb8 FREE DDT_FREE - - e8dc400 FREE DDT_FREE - - e91f9b8 FREE DDT_FREE - - e8f8910 FREE DDT_FREE - - d15e440 FREE DDT_FREE - - d2706d8 FREE DDT_FREE - - cde2950 FREE DDT_FREE - - e8e8138 FREE DDT_FREE - - e8f0010 FREE DDT_FREE - - e8f2310 FREE DDT_FREE - - e8d53d8 FREE DDT_FREE - - e8fa918 FREE DDT_FREE - - e91fca8 FREE DDT_FREE - - e8f6320 FREE DDT_FREE - - e8efbc8 FREE DDT_FREE - - d14e6d0 FREE DDT_FREE - - e90b360 FREE DDT_FREE - - e8f7618 FREE DDT_FREE - - e937cf0 FREE DDT_FREE - - e8eebc0 FREE DDT_FREE - - d28b8e0 FREE DDT_FREE - - e90f088 FREE DDT_FREE - - d29a070 FREE DDT_FREE - - d294918 FREE DDT_FREE - - e8e4120 FREE DDT_FREE - - e90a938 FREE DDT_FREE - - e8e49f0 FREE DDT_FREE - - d15d438 FREE DDT_FREE - - d13c610 FREE DDT_FREE - - 8791960 FREE DDT_FREE - - e8f7038 FREE DDT_FREE - - d2719d0 FREE DDT_FREE - - e8f1bd8 FREE DDT_FREE - - d295630 FREE DDT_FREE - - e8e59f8 FREE DDT_FREE - - e92c118 FREE DDT_FREE - - d3e1658 FREE DDT_FREE - - e8f2020 FREE DDT_FREE - - e8e5708 FREE DDT_FREE - - e8e7710 FREE DDT_FREE - - e8eba18 FREE DDT_FREE - - e911090 FREE DDT_FREE - - 894c340 FREE DDT_FREE - - e8e3cd8 FREE DDT_FREE - - d2cd9a0 FREE DDT_FREE - - d13c030 FREE DDT_FREE - - e8f2600 FREE DDT_FREE - - e8fb920 FREE DDT_FREE - - d295050 FREE DDT_FREE - - 8746960 NULL OPEN - - 856abc8 NULL OPEN - - e8fd638 NULL OPEN - -
From the corefile we see this:
> ::stacks -c zio_execute
THREAD STATE SOBJ COUNT
f0 PARKED CV 95
libc.so.1`cond_wait_queue+0x60
libc.so.1`__cond_wait+0x86
libc.so.1`cond_wait+0x24
libzpool.so.1`cv_wait+0x40
libzpool.so.1`ddt_lookup+0xc6
libzpool.so.1`zio_ddt_free+0x5c
libzpool.so.1`zio_execute+0x13b
libzpool.so.1`taskq_thread+0x9c
libc.so.1`_thrp_setup+0x9b
libc.so.1`_lwp_start
137 PARKED CV 1
libc.so.1`cond_wait_queue+0x60
libc.so.1`__cond_wait+0x86
libc.so.1`cond_wait+0x24
libzpool.so.1`cv_wait+0x40
libzpool.so.1`zio_wait+0x7f
libzpool.so.1`dbuf_read+0x27f
libzpool.so.1`dmu_buf_hold+0xae
libzpool.so.1`zap_get_leaf_byblk+0x87
libzpool.so.1`zap_deref_leaf+0xa4
libzpool.so.1`fzap_length+0x31
libzpool.so.1`zap_length_uint64+0x62
libzpool.so.1`ddt_zap_lookup+0x3e
libzpool.so.1`ddt_object_lookup+0x5a
libzpool.so.1`ddt_lookup+0x14b
libzpool.so.1`zio_ddt_free+0x5c
libzpool.so.1`zio_execute+0x13b
libzpool.so.1`taskq_thread+0x9c
libc.so.1`_thrp_setup+0x9b
libc.so.1`_lwp_start
We have a bunch of threads that are waiting for the ddt_load to finish to
complete and another set that are trying to load the ddt. Unfortunately we're
out of taskqs to do the work. This is because in vdev_file_io_start() we use
the ZIO_TYPE_FREE issue taskq to do asynchronous I/O but it can't complete the
I/O because all the FREE issue threads are already consumed.
Switching vdev_file_io_start() to use the system_taskq instead.
Updated by Christopher Siden over 10 years ago
- Status changed from In Progress to Closed
commit 2c1e2b4
Author: George Wilson <george.wilson@delphix.com>
Date: Wed Aug 7 11:24:34 2013
3949 ztest fault injection should avoid resilvering devices
3950 ztest: deadman fires when we're doing a scan
3951 ztest hang when running dedup test
3952 ztest: ztest_reguid test and ztest_fault_inject don't place nice together
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Actions