Bug #5693
closedztest fails in dbuf_verify: buf[i] == 0, due to dedup and bp_override
100%
Description
sometimes ztest fails with:
::status
debugging core file of ztest (32-bit) from os-ztest-373.dcenter
file: /usr/bin/i86/ztest
initial argv: /usr/bin/i86/ztest
threading model: native threads
status: process terminated by SIGABRT (Abort), pid=103191 uid=65433 code=-1
panic message: assertion failed for thread 0xf09aba40, thread-id 544: buf[i] == 0, file ../../../uts/common/fs/zfs/dbuf.c, line 415
::stack
libc.so.1`_lwp_kill+0x15(ebec3410, ebec3410, 75, ebec3481, fea99708, 0)
libc.so.1`_assfail+0x1b9(fedcdcbb, fedb5aec, 19f, 89abcdef, ebec37b8)
libc.so.1`assfail+0x2b(fedcdcbb, fedb5aec, 19f, fef5b000)
libzpool.so.1`dbuf_verify+0x77b(9cd3440, 0)
libzpool.so.1`dbuf_rele_and_unlock+0x54(9cd3440, 28, 8c11340, 1)
libzpool.so.1`dbuf_write_done+0x70b(8e14ce0, 0, 9cd3440, fed822e2)
libzpool.so.1`dbuf_write_override_done+0x18b(8e14ce0, 3, 0, 8dfcc6c)
libzpool.so.1`zio_done+0xabf(8e14ce0, 0, ebec3a08, feda30d5)
libzpool.so.1`zio_execute+0x1a2(8e14ce0, 0, 100, feda719a)
libzpool.so.1`zio_nowait+0xc9(8e14ce0, 9b5f6a8, 8c11340, feedf348)
libzpool.so.1`dbuf_sync_leaf+0x5e5(9cf4b00, 8c11340, 8, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x8b(9cf4844, 8c11340, 8c11340, feedf348, 9cf492c, 0)
libzpool.so.1`dbuf_sync_indirect+0x313(9cf4800, 8c11340, 0, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x7a(9cf4944, 8c11340, 8c11340, feedf348, 98cde2c, 0)
libzpool.so.1`dbuf_sync_indirect+0x313(9cf4900, 8c11340, 0, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x7a(98cde44, 8c11340, 8c11340, fed39f0b, 9b0a048, fedb300c)
libzpool.so.1`dbuf_sync_indirect+0x313(98cde00, 8c11340, 914e400, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x7a(9b0a148, 8c11340, 200, fed1d176)
libzpool.so.1`dnode_sync+0x89d(9b0a048, 8c11340, 0, fed2c8ea)
libzpool.so.1`dmu_objset_sync_dnodes+0x11a(903f1b4, 0, 8c11340, ebec3db0, 8de23a8, 0)
libzpool.so.1`dmu_objset_sync+0x2c1(903ef80, 8e06cd8, 8c11340, 80bf888)
libzpool.so.1`dsl_dataset_sync+0xc3(903f2c0, 8e06cd8, 8c11340, fed695da)
libzpool.so.1`dsl_pool_sync+0xce(80bf640, 28, 0, 0)
libzpool.so.1`spa_sync+0x516(8721000, 28, 0, 241, f09aba40, fef61b80)
libzpool.so.1`txg_sync_thread+0x41c(80bf640, 0, 0, 0)
libc.so.1`_thrp_setup+0x88(f09aba40)
libc.so.1`_lwp_start(f09aba40, 0, 0, 0, 0, 0)
8e14ce0::print zio_t io_prop
io_prop = {
io_prop.zp_checksum = 8 (ZIO_CHECKSUM_SHA256)
io_prop.zp_compress = 2 (ZIO_COMPRESS_OFF)
io_prop.zp_type = 0t26 (DMU_OT_UINT64_OTHER)
io_prop.zp_level = 0
io_prop.zp_copies = 0x1
io_prop.zp_dedup = 0x1 (B_TRUE)
io_prop.zp_dedup_verify = 0 (0)
io_prop.zp_nopwrite = 0 (0)
}
9cd3440::print dmu_buf_impl_t db.db_data|::print ztest_block_tag_t
{
bt_magic = 0x123456789abcdef
bt_objset = 0x28
bt_object = 0x1a
bt_offset = 0x40000
bt_gen = 0
bt_txg = 0x28
bt_crtxg = 0x28
}
Note that the i/o in question was an override (we've called dbuf_write_override_done()), and we wanted to dedup (zp_dedup is set). The dbuf's data is filled in (looks like a normal ztest_block_tag_t), but we expect it to be all zeroed, because we have written it, it isn't dirty anymore, and the BP is a hole.
Looking at zio_write_bp_init(), if we are doing io_bp_override, then we set the pipeline to ZIO_INTERLOCK_PIPELINE, meaning that we don't do any i/o for it. If we are trying to do dedup, and the checksum function of the override BP is the checksum type we are using for dedup, then we add in the DDT_WRITE stage, which will bump the refcount of this block in the DDT. However, if we are trying to do dedup, and the checksum function is not the same (because the checksum property changed?), then we zero out the BP. I'm not sure what the intent was here (it's done this since dedup was implemented), but the result is that we insert a hole, transforming the user's data to zeros.
Related issues
Updated by Electric Monk over 7 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit 7f7ace370074e350853da254c65688fd43ddc695
commit 7f7ace370074e350853da254c65688fd43ddc695 Author: Matthew Ahrens <mahrens@delphix.com> Date: 2015-03-24T16:49:04.000Z 5693 ztest fails in dbuf_verify: buf[i] == 0, due to dedup and bp_override Reviewed by: George Wilson <george@delphix.com> Reviewed by: Christopher Siden <christopher.siden@delphix.com> Reviewed by: Bayard Bell <buffer.g.overflow@gmail.com> Approved by: Dan McDonald <danmcd@omniti.com>
Updated by Marcel Telka over 2 years ago
- Related to Bug #6319: assertion failed in zio_ddt_write: bp->blk_birth == txg added