illumos gate

In dbuf_dirty(), we need to grab the dn_struct_rwlock before looking at the db_blkptr, to prevent it from being changed by syncing context.

Otherwise we may see that ztest got a segfault from this stack:

libzpool.so.1`dva_get_dsize_sync+0x98(872f000, b32b240, fed7811b, 0, b4cda20, 0
)
libzpool.so.1`bp_get_dsize+0x60(872f000, b32b240, 0, 97cb780, 9d4c1a8, 0)
libzpool.so.1`dbuf_dirty+0x9b3(ce0a100, 97cb780, 9, fecd2530)
libzpool.so.1`dmu_buf_will_dirty+0xc3(ce0a100, 97cb780, ea293d6c, 1)
libzpool.so.1`zap_lockdir+0x1a0(8aaa3c0, 1, 0, 97cb780, 1, 1)
libzpool.so.1`zap_remove_norm+0x30(8aaa3c0, 1, 0, 8728b10, 0, 97cb780)
libzpool.so.1`zap_remove+0x29(8aaa3c0, 1, 0, 8728b10, 97cb780, a)
ztest_replay_remove+0x225(ea294588, 8728ae8, 0, 38010000, 0, 0)
ztest_remove+0x9f(ea294588, ea293f50, 4, 3)
ztest_object_init+0x78(ea294588, ea293f50, 4e0, 1)
ztest_dmu_object_alloc_free+0x71(ea294588, 13)
ztest_dmu_objset_create_destroy+0x224(80cef08, 13, 0, 805d36c, 9017ad44, 0)
ztest_execute+0x89(a, 807c720, 13, 0)
ztest_thread+0xea(13, 0, 0, 0)
libc.so.1`_thrp_setup+0x88(f0983240)
libc.so.1`_lwp_start(f0983240, 0, 0, 0, 0, 0)

Looking into it a bit, we see that this is an embedded blockpointer, so BP_GET_NDVAS should have returned 0:

b32b240::blkptr

EMBEDDED [L0 ZAP_OTHER] et=0 LZ4 size=200L/4aP birth=80L

Instead, it looks like another thread is modifying this blockpointer:

b32b240::ugrep | ::whatis

f47a0e0c is in [ stack tid=0x19f ]
ebd6ec40 is in [ stack tid=0x226 ]
ea293bd0 is in [ stack tid=0x244 ]
ea293be4 is in [ stack tid=0x244 ]
ea293c54 is in [ stack tid=0x244 ]
ea293c7c is in [ stack tid=0x244 ]
ce0a134 is ce0a100+34, allocated from dmu_buf_impl_t:
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ce08478 ce0a100 5a8563be90 580
810a290 80913e8 80a26f8
libumem.so.1`umem_cache_alloc_debug+0x1fe
libumem.so.1`umem_cache_alloc+0x99
libzpool.so.1`dbuf_create+0xa5
libzpool.so.1`dbuf_hold_impl+0x219
libzpool.so.1`dbuf_hold_level+0x31
libzpool.so.1`dbuf_hold+0x26
libzpool.so.1`dmu_buf_hold_noread+0x6d
libzpool.so.1`dmu_buf_hold+0x40
libzpool.so.1`zap_lockdir+0x44
libzpool.so.1`zap_remove_norm+0x30
libzpool.so.1`zap_remove+0x29
ztest_replay_remove+0x225
ztest_remove+0x9f
ztest_object_init+0x78
ztest_dmu_object_alloc_free+0x71

0x19f::findstack -v

stack pointer for thread 19f: f47a0a68
[ f47a0a68 libzpool.so.1`range_tree_verify+0x1b() ]
f47a0ac8 libzpool.so.1`metaslab_check_free_impl+0x1ef(8756140, 205ce000, 0, 2000, 0, 7f)
f47a0b28 libzpool.so.1`metaslab_free_concrete+0x2fc(8756140, 205ce000, 0, 2000, 0, 50)
f47a0b88 libzpool.so.1`metaslab_free_impl+0x120(8756140, 205ce000, 0, 2000, 0, 50)
f47a0be8 libzpool.so.1`metaslab_free_dva+0x114(872f000, cbd0d2c, 50, 0)
f47a0c48 libzpool.so.1`metaslab_free+0x181(872f000, cbd0d2c, 50, 0, 0, 872f000)
f47a0c78 libzpool.so.1`zio_dva_free+0x29(cbd0cc8, 0, 200, fed66bd8)
f47a0cc8 libzpool.so.1`zio_execute+0x1a2(cbd0cc8, 801b, 50, fed66e52, 0, 8756140)
f47a0d08 libzpool.so.1`zio_wait+0xc5(cbd0cc8, 0, f47a0d68, fea2b9db, 50, 0)
f47a0d48 libzpool.so.1`zio_free+0xd5(872f000, 50, 0, e008abc)
f47a0d68 libzpool.so.1`dsl_free+0x26(80c0640, 50, 0, e008abc, b4cd948, f897ca40)
f47a0dc8 libzpool.so.1`dsl_dataset_block_kill+0x320(90e6840, e008abc, 95ca500)
f47a0e28 libzpool.so.1`dbuf_write_done+0x209(e008998, ce14a40)
f47a0e88 libzpool.so.1`arc_write_done+0x659(e008998, 3, 0, fed6a64e)
f47a0f18 libzpool.so.1`zio_done+0xbe2(e008998, 0, 0, fed66bd8)
f47a0f68 libzpool.so.1`zio_execute+0x1a2(e008998)
f47a0fc8 libcmdutils.so.1`taskq_thread+0x1c6(873ff10, 0, 0, 0)
f47a0fe8 libc.so.1`_thrp_setup+0x88(f897ca40)
f47a0ff8 libc.so.1`_lwp_start(f897ca40, 0, 0, 0, 0, 0)