illumos gate

sometimes ztest fails with:

::status

debugging core file of ztest (32-bit) from os-ztest-373.dcenter
file: /usr/bin/i86/ztest
initial argv: /usr/bin/i86/ztest
threading model: native threads
status: process terminated by SIGABRT (Abort), pid=103191 uid=65433 code=-1
panic message: assertion failed for thread 0xf09aba40, thread-id 544: buf[i] == 0, file ../../../uts/common/fs/zfs/dbuf.c, line 415

::stack

libc.so.1`_lwp_kill+0x15(ebec3410, ebec3410, 75, ebec3481, fea99708, 0)
libc.so.1`_assfail+0x1b9(fedcdcbb, fedb5aec, 19f, 89abcdef, ebec37b8)
libc.so.1`assfail+0x2b(fedcdcbb, fedb5aec, 19f, fef5b000)
libzpool.so.1`dbuf_verify+0x77b(9cd3440, 0)
libzpool.so.1`dbuf_rele_and_unlock+0x54(9cd3440, 28, 8c11340, 1)
libzpool.so.1`dbuf_write_done+0x70b(8e14ce0, 0, 9cd3440, fed822e2)
libzpool.so.1`dbuf_write_override_done+0x18b(8e14ce0, 3, 0, 8dfcc6c)
libzpool.so.1`zio_done+0xabf(8e14ce0, 0, ebec3a08, feda30d5)
libzpool.so.1`zio_execute+0x1a2(8e14ce0, 0, 100, feda719a)
libzpool.so.1`zio_nowait+0xc9(8e14ce0, 9b5f6a8, 8c11340, feedf348)
libzpool.so.1`dbuf_sync_leaf+0x5e5(9cf4b00, 8c11340, 8, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x8b(9cf4844, 8c11340, 8c11340, feedf348, 9cf492c, 0)
libzpool.so.1`dbuf_sync_indirect+0x313(9cf4800, 8c11340, 0, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x7a(9cf4944, 8c11340, 8c11340, feedf348, 98cde2c, 0)
libzpool.so.1`dbuf_sync_indirect+0x313(9cf4900, 8c11340, 0, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x7a(98cde44, 8c11340, 8c11340, fed39f0b, 9b0a048, fedb300c)
libzpool.so.1`dbuf_sync_indirect+0x313(98cde00, 8c11340, 914e400, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x7a(9b0a148, 8c11340, 200, fed1d176)
libzpool.so.1`dnode_sync+0x89d(9b0a048, 8c11340, 0, fed2c8ea)
libzpool.so.1`dmu_objset_sync_dnodes+0x11a(903f1b4, 0, 8c11340, ebec3db0, 8de23a8, 0)
libzpool.so.1`dmu_objset_sync+0x2c1(903ef80, 8e06cd8, 8c11340, 80bf888)
libzpool.so.1`dsl_dataset_sync+0xc3(903f2c0, 8e06cd8, 8c11340, fed695da)
libzpool.so.1`dsl_pool_sync+0xce(80bf640, 28, 0, 0)
libzpool.so.1`spa_sync+0x516(8721000, 28, 0, 241, f09aba40, fef61b80)
libzpool.so.1`txg_sync_thread+0x41c(80bf640, 0, 0, 0)
libc.so.1`_thrp_setup+0x88(f09aba40)
libc.so.1`_lwp_start(f09aba40, 0, 0, 0, 0, 0)

8e14ce0::print zio_t io_prop

io_prop = {
io_prop.zp_checksum = 8 (ZIO_CHECKSUM_SHA256)
io_prop.zp_compress = 2 (ZIO_COMPRESS_OFF)
io_prop.zp_type = 0t26 (DMU_OT_UINT64_OTHER)
io_prop.zp_level = 0
io_prop.zp_copies = 0x1
io_prop.zp_dedup = 0x1 (B_TRUE)
io_prop.zp_dedup_verify = 0 (0)
io_prop.zp_nopwrite = 0 (0)
}

9cd3440::print dmu_buf_impl_t db.db_data|::print ztest_block_tag_t

{
bt_magic = 0x123456789abcdef
bt_objset = 0x28
bt_object = 0x1a
bt_offset = 0x40000
bt_gen = 0
bt_txg = 0x28
bt_crtxg = 0x28
}

Note that the i/o in question was an override (we've called dbuf_write_override_done()), and we wanted to dedup (zp_dedup is set). The dbuf's data is filled in (looks like a normal ztest_block_tag_t), but we expect it to be all zeroed, because we have written it, it isn't dirty anymore, and the BP is a hole.
Looking at zio_write_bp_init(), if we are doing io_bp_override, then we set the pipeline to ZIO_INTERLOCK_PIPELINE, meaning that we don't do any i/o for it. If we are trying to do dedup, and the checksum function of the override BP is the checksum type we are using for dedup, then we add in the DDT_WRITE stage, which will bump the refcount of this block in the DDT. However, if we are trying to do dedup, and the checksum function is not the same (because the checksum property changed?), then we zero out the BP. I'm not sure what the intent was here (it's done this since dedup was implemented), but the result is that we insert a hole, transforming the user's data to zeros.