Project

General

Profile

Bug #7147

ztest: ztest_ddt_repair fails with ztest_pattern_match assertion

Added by Matthew Ahrens about 3 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
zfs - Zettabyte File System
Start date:
2016-06-28
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

Here's the dbuf we're currently reading:

966f200::dbuf

addr object lvl blkid holds os
966f200 4 0 0 1 ztest/ds_3

966f200::print dmu_buf_t db_data

db_data = 0x9ae0400

0x9ae0400/10J

0x9ae0400: c1c7ced932020d c1c7ced932020d c1c7ced932020d c1c7ced932020d
c1c7ced932020d c1c7ced932020d c1c7ced932020d c1c7ced932020d
c1c7ced932020d c1c7ced932020d
The pattern we're expecting is actually this: a34ae10b5f2db2. If we attempt to read the block on disk we find that it has matches what ztest_ddt_repair() would have written:

~c1c7ced932020d=J

ff3e383126cdfdf2

966f200::print dmu_buf_impl_t db_blkptr | ::blkptr

DVA0=<0:71d3c00:800>
[L0 UINT64_OTHER] SHA256 OFF LE contiguous dedup single
size=400L/400P birth=55L/55P fill=1
cksum=18486450d3ce8c6d:75a72f4bbf117b0f:2d3a226314eb5650:2eb0fd68648b1af0

  1. zdb -U /rpool/tmp/zpool.cache -R ztest 0:71d3c00:800 | head
    Found vdev type: mirror
0:71d3c00:800
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
000000: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000010: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000020: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000030: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000040: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000050: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000060: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
Since this is a dedup-dittoed block let's look at the other copy on disk:
  1. zdb -U /rpool/tmp/zpool.cache -R ztest 0:71d5000:800 | head
    Found vdev type: mirror

0:71d5000:800
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
000000: 00c1c7ced932020d 00c1c7ced932020d ..2.......2.....
000010: 00c1c7ced932020d 00c1c7ced932020d ..2.......2.....
000020: 00c1c7ced932020d 00c1c7ced932020d ..2.......2.....
000030: 00c1c7ced932020d 00c1c7ced932020d ..2.......2.....
000040: 00c1c7ced932020d 00c1c7ced932020d ..2.......2.....
000050: 00c1c7ced932020d 00c1c7ced932020d ..2.......2.....
000060: 00c1c7ced932020d 00c1c7ced932020d ..2.......2.....
Hmm, this matches what's in our dbuf's data buffer so it explains where we got the data. So far everything is as we expect. Let's see how we derived the pattern we're expecting.
According to zdb the root dataset's fsid_guid value is 0x7073e79c2baab. If we calculate what our fsid_guid should be then we derive this:

7073e79c2baab^c1c7ced932020d=J

c6c0f0a0f0b8a6
When we look at the fsid_guid for the dsl dataset associated with the dbuf we get this:

966f200::print dmu_buf_impl_t db_objset->os_dsl_dataset->ds_fsid_guid

db_objset->os_dsl_dataset->ds_fsid_guid = 0xa44ddf729d9719
If I calculate the pattern based on the two fsid_guid values we have we get the following:

0xa44ddf729d9719^7073e79c2baab=J

a34ae10b5f2db2
This matches the pattern that we're expecting. So it seems that somehow the DDT had it's dedup ditto-ed block corrupted somehow.

It turns out that it's possible for a dataset to find that its ds_fsid_guid is still in use and so it must pick a new one. This breaks the ztest_ddt_repair() test since it's expecting the ds_fsid_guid to be the same throughout the lifetime of the dataset. We can find an already in-use ds_fsid_guid if the dataset is being destroyed but has not yet been processed by the dbu_evict_taskq. To resolve this we will use the ds_guid which should never change to create the pattern used by ztest_ddt_repair().

History

#1

Updated by Electric Monk about 3 years ago

  • % Done changed from 0 to 100
  • Status changed from New to Closed

git commit aab80726335c76a7cae32c7300890248d73a51e3

commit  aab80726335c76a7cae32c7300890248d73a51e3
Author: George Wilson <george.wilson@delphix.com>
Date:   2016-07-19T23:28:59.000Z

    7147 ztest: ztest_ddt_repair fails with ztest_pattern_match assertion
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: Prakash Surya <prakash.surya@delphix.com>
    Approved by: Robert Mustacchi <rm@joyent.com>

Also available in: Atom PDF