Project

General

Profile

Bug #5693

ztest fails in dbuf_verify: buf[i] == 0, due to dedup and bp_override

Added by Matthew Ahrens over 4 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Category:
zfs - Zettabyte File System
Start date:
2015-03-08
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

sometimes ztest fails with:

::status

debugging core file of ztest (32-bit) from os-ztest-373.dcenter
file: /usr/bin/i86/ztest
initial argv: /usr/bin/i86/ztest
threading model: native threads
status: process terminated by SIGABRT (Abort), pid=103191 uid=65433 code=-1
panic message: assertion failed for thread 0xf09aba40, thread-id 544: buf[i] == 0, file ../../../uts/common/fs/zfs/dbuf.c, line 415

::stack

libc.so.1`_lwp_kill+0x15(ebec3410, ebec3410, 75, ebec3481, fea99708, 0)
libc.so.1`_assfail+0x1b9(fedcdcbb, fedb5aec, 19f, 89abcdef, ebec37b8)
libc.so.1`assfail+0x2b(fedcdcbb, fedb5aec, 19f, fef5b000)
libzpool.so.1`dbuf_verify+0x77b(9cd3440, 0)
libzpool.so.1`dbuf_rele_and_unlock+0x54(9cd3440, 28, 8c11340, 1)
libzpool.so.1`dbuf_write_done+0x70b(8e14ce0, 0, 9cd3440, fed822e2)
libzpool.so.1`dbuf_write_override_done+0x18b(8e14ce0, 3, 0, 8dfcc6c)
libzpool.so.1`zio_done+0xabf(8e14ce0, 0, ebec3a08, feda30d5)
libzpool.so.1`zio_execute+0x1a2(8e14ce0, 0, 100, feda719a)
libzpool.so.1`zio_nowait+0xc9(8e14ce0, 9b5f6a8, 8c11340, feedf348)
libzpool.so.1`dbuf_sync_leaf+0x5e5(9cf4b00, 8c11340, 8, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x8b(9cf4844, 8c11340, 8c11340, feedf348, 9cf492c, 0)
libzpool.so.1`dbuf_sync_indirect+0x313(9cf4800, 8c11340, 0, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x7a(9cf4944, 8c11340, 8c11340, feedf348, 98cde2c, 0)
libzpool.so.1`dbuf_sync_indirect+0x313(9cf4900, 8c11340, 0, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x7a(98cde44, 8c11340, 8c11340, fed39f0b, 9b0a048, fedb300c)
libzpool.so.1`dbuf_sync_indirect+0x313(98cde00, 8c11340, 914e400, fed1d0ed)
libzpool.so.1`dbuf_sync_list+0x7a(9b0a148, 8c11340, 200, fed1d176)
libzpool.so.1`dnode_sync+0x89d(9b0a048, 8c11340, 0, fed2c8ea)
libzpool.so.1`dmu_objset_sync_dnodes+0x11a(903f1b4, 0, 8c11340, ebec3db0, 8de23a8, 0)
libzpool.so.1`dmu_objset_sync+0x2c1(903ef80, 8e06cd8, 8c11340, 80bf888)
libzpool.so.1`dsl_dataset_sync+0xc3(903f2c0, 8e06cd8, 8c11340, fed695da)
libzpool.so.1`dsl_pool_sync+0xce(80bf640, 28, 0, 0)
libzpool.so.1`spa_sync+0x516(8721000, 28, 0, 241, f09aba40, fef61b80)
libzpool.so.1`txg_sync_thread+0x41c(80bf640, 0, 0, 0)
libc.so.1`_thrp_setup+0x88(f09aba40)
libc.so.1`_lwp_start(f09aba40, 0, 0, 0, 0, 0)

8e14ce0::print zio_t io_prop

io_prop = {
io_prop.zp_checksum = 8 (ZIO_CHECKSUM_SHA256)
io_prop.zp_compress = 2 (ZIO_COMPRESS_OFF)
io_prop.zp_type = 0t26 (DMU_OT_UINT64_OTHER)
io_prop.zp_level = 0
io_prop.zp_copies = 0x1
io_prop.zp_dedup = 0x1 (B_TRUE)
io_prop.zp_dedup_verify = 0 (0)
io_prop.zp_nopwrite = 0 (0)
}

9cd3440::print dmu_buf_impl_t db.db_data|::print ztest_block_tag_t

{
bt_magic = 0x123456789abcdef
bt_objset = 0x28
bt_object = 0x1a
bt_offset = 0x40000
bt_gen = 0
bt_txg = 0x28
bt_crtxg = 0x28
}

Note that the i/o in question was an override (we've called dbuf_write_override_done()), and we wanted to dedup (zp_dedup is set). The dbuf's data is filled in (looks like a normal ztest_block_tag_t), but we expect it to be all zeroed, because we have written it, it isn't dirty anymore, and the BP is a hole.
Looking at zio_write_bp_init(), if we are doing io_bp_override, then we set the pipeline to ZIO_INTERLOCK_PIPELINE, meaning that we don't do any i/o for it. If we are trying to do dedup, and the checksum function of the override BP is the checksum type we are using for dedup, then we add in the DDT_WRITE stage, which will bump the refcount of this block in the DDT. However, if we are trying to do dedup, and the checksum function is not the same (because the checksum property changed?), then we zero out the BP. I'm not sure what the intent was here (it's done this since dedup was implemented), but the result is that we insert a hole, transforming the user's data to zeros.

History

#1

Updated by Electric Monk over 4 years ago

  • % Done changed from 0 to 100
  • Status changed from New to Closed

git commit 7f7ace370074e350853da254c65688fd43ddc695

commit  7f7ace370074e350853da254c65688fd43ddc695
Author: Matthew Ahrens <mahrens@delphix.com>
Date:   2015-03-24T16:49:04.000Z

    5693 ztest fails in dbuf_verify: buf[i] == 0, due to dedup and bp_override
    Reviewed by: George Wilson <george@delphix.com>
    Reviewed by: Christopher Siden <christopher.siden@delphix.com>
    Reviewed by: Bayard Bell <buffer.g.overflow@gmail.com>
    Approved by: Dan McDonald <danmcd@omniti.com>

Also available in: Atom PDF