Bug #5693


ztest fails in dbuf_verify: buf[i] == 0, due to dedup and bp_override

Added by Matthew Ahrens over 7 years ago. Updated over 7 years ago.

zfs - Zettabyte File System
Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


sometimes ztest fails with:


debugging core file of ztest (32-bit) from os-ztest-373.dcenter
file: /usr/bin/i86/ztest
initial argv: /usr/bin/i86/ztest
threading model: native threads
status: process terminated by SIGABRT (Abort), pid=103191 uid=65433 code=-1
panic message: assertion failed for thread 0xf09aba40, thread-id 544: buf[i] == 0, file ../../../uts/common/fs/zfs/dbuf.c, line 415

::stack`_lwp_kill+0x15(ebec3410, ebec3410, 75, ebec3481, fea99708, 0)`_assfail+0x1b9(fedcdcbb, fedb5aec, 19f, 89abcdef, ebec37b8)`assfail+0x2b(fedcdcbb, fedb5aec, 19f, fef5b000)`dbuf_verify+0x77b(9cd3440, 0)`dbuf_rele_and_unlock+0x54(9cd3440, 28, 8c11340, 1)`dbuf_write_done+0x70b(8e14ce0, 0, 9cd3440, fed822e2)`dbuf_write_override_done+0x18b(8e14ce0, 3, 0, 8dfcc6c)`zio_done+0xabf(8e14ce0, 0, ebec3a08, feda30d5)`zio_execute+0x1a2(8e14ce0, 0, 100, feda719a)`zio_nowait+0xc9(8e14ce0, 9b5f6a8, 8c11340, feedf348)`dbuf_sync_leaf+0x5e5(9cf4b00, 8c11340, 8, fed1d0ed)`dbuf_sync_list+0x8b(9cf4844, 8c11340, 8c11340, feedf348, 9cf492c, 0)`dbuf_sync_indirect+0x313(9cf4800, 8c11340, 0, fed1d0ed)`dbuf_sync_list+0x7a(9cf4944, 8c11340, 8c11340, feedf348, 98cde2c, 0)`dbuf_sync_indirect+0x313(9cf4900, 8c11340, 0, fed1d0ed)`dbuf_sync_list+0x7a(98cde44, 8c11340, 8c11340, fed39f0b, 9b0a048, fedb300c)`dbuf_sync_indirect+0x313(98cde00, 8c11340, 914e400, fed1d0ed)`dbuf_sync_list+0x7a(9b0a148, 8c11340, 200, fed1d176)`dnode_sync+0x89d(9b0a048, 8c11340, 0, fed2c8ea)`dmu_objset_sync_dnodes+0x11a(903f1b4, 0, 8c11340, ebec3db0, 8de23a8, 0)`dmu_objset_sync+0x2c1(903ef80, 8e06cd8, 8c11340, 80bf888)`dsl_dataset_sync+0xc3(903f2c0, 8e06cd8, 8c11340, fed695da)`dsl_pool_sync+0xce(80bf640, 28, 0, 0)`spa_sync+0x516(8721000, 28, 0, 241, f09aba40, fef61b80)`txg_sync_thread+0x41c(80bf640, 0, 0, 0)`_thrp_setup+0x88(f09aba40)`_lwp_start(f09aba40, 0, 0, 0, 0, 0)

8e14ce0::print zio_t io_prop

io_prop = {
io_prop.zp_checksum = 8 (ZIO_CHECKSUM_SHA256)
io_prop.zp_compress = 2 (ZIO_COMPRESS_OFF)
io_prop.zp_type = 0t26 (DMU_OT_UINT64_OTHER)
io_prop.zp_level = 0
io_prop.zp_copies = 0x1
io_prop.zp_dedup = 0x1 (B_TRUE)
io_prop.zp_dedup_verify = 0 (0)
io_prop.zp_nopwrite = 0 (0)

9cd3440::print dmu_buf_impl_t db.db_data|::print ztest_block_tag_t

bt_magic = 0x123456789abcdef
bt_objset = 0x28
bt_object = 0x1a
bt_offset = 0x40000
bt_gen = 0
bt_txg = 0x28
bt_crtxg = 0x28

Note that the i/o in question was an override (we've called dbuf_write_override_done()), and we wanted to dedup (zp_dedup is set). The dbuf's data is filled in (looks like a normal ztest_block_tag_t), but we expect it to be all zeroed, because we have written it, it isn't dirty anymore, and the BP is a hole.
Looking at zio_write_bp_init(), if we are doing io_bp_override, then we set the pipeline to ZIO_INTERLOCK_PIPELINE, meaning that we don't do any i/o for it. If we are trying to do dedup, and the checksum function of the override BP is the checksum type we are using for dedup, then we add in the DDT_WRITE stage, which will bump the refcount of this block in the DDT. However, if we are trying to do dedup, and the checksum function is not the same (because the checksum property changed?), then we zero out the BP. I'm not sure what the intent was here (it's done this since dedup was implemented), but the result is that we insert a hole, transforming the user's data to zeros.

Related issues

Related to illumos gate - Bug #6319: assertion failed in zio_ddt_write: bp->blk_birth == txgClosedMatthew Ahrens2015-10-10

Actions #1

Updated by Electric Monk over 7 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 7f7ace370074e350853da254c65688fd43ddc695

commit  7f7ace370074e350853da254c65688fd43ddc695
Author: Matthew Ahrens <>
Date:   2015-03-24T16:49:04.000Z

    5693 ztest fails in dbuf_verify: buf[i] == 0, due to dedup and bp_override
    Reviewed by: George Wilson <>
    Reviewed by: Christopher Siden <>
    Reviewed by: Bayard Bell <>
    Approved by: Dan McDonald <>

Actions #2

Updated by Marcel Telka over 2 years ago

  • Related to Bug #6319: assertion failed in zio_ddt_write: bp->blk_birth == txg added

Also available in: Atom PDF