Project

General

Profile

Bug #7181

race between zfs_mount and zfs_ioc_rollback

Added by Andriy Gapon about 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
zfs - Zettabyte File System
Start date:
2016-07-12
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

zfsvfs_setup() is called in both zfs_mount and zfs_resume_fs paths.
dmu_objset_set_user(zfsvfs->z_os, zfsvfs) is called early in zfsvfs_setup() before the setup is actually completed,
thus an under-constructed zfsvfs becomes visible.
Additionally, there is nothing to serialize the two call paths. As a result two threads can step on each other's toes.

assertion failed: zilog->zl_clean_taskq == NULL, file:
../../common/fs/zfs/zil.c, line: 1772

> $c
vpanic()
0xfffffffffbdf6928()
zil_open+0x45(ffffff1bbc5dd000, fffffffff7993880)
zfsvfs_setup+0x84(ffffffb378d77000, 0)
zfs_resume_fs+0x132(ffffffb378d77000, ffffffb37ddcf000)
zfs_ioc_rollback+0x96(ffffffb37ddcf000, ffffff01dcdc4cd0, ffffff01aa091000)
zfsdev_ioctl+0x215(10a00000000, 5a19, 80465f8, 100003, ffffff01ab318368,
ffffff0004b59e58)
cdev_ioctl+0x39(10a00000000, 5a19, 80465f8, 100003, ffffff01ab318368,
ffffff0004b59e58)
spec_ioctl+0x60(ffffff0197737700, 5a19, 80465f8, 100003,
ffffff01ab318368, ffffff0004b59e58)
fop_ioctl+0x55(ffffff0197737700, 5a19, 80465f8, 100003,
ffffff01ab318368, ffffff0004b59e58)
ioctl+0x9b(7, 5a19, 80465f8)
sys_syscall32+0x1f7()

> ffffff1bbc5dd000::print objset_t os_zil
os_zil = 0xffffff1c053cf7c0
> 0xffffff1c053cf7c0::print zilog_t zl_clean_taskq
zl_clean_taskq = 0xffffff01ce8a3e98
> 0xffffff01ce8a3e98::whatis
ffffff01ce8a3e98 is allocated from taskq_cache:
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
ffffff01a4d84cd0 ffffff01ce8a3e98    8493ac8d95d2e ffffff01b6123460
                 ffffff0185faf008 ffffff01819cfb40 ffffff0184b68890
                 kmem_cache_alloc_debug+0x2e0
                 kmem_cache_alloc+0xdd
                 taskq_create_common+0x63
                 taskq_create+0xa0
                 zil_open+0xb9
                 zfsvfs_setup+0x84
                 zfs_domount+0x19f
                 zfs_mount+0x24f
                 fsop_mount+0x1e
                 domount+0x9f4
                 mount+0x167
                 syscall_ap+0x94

Let's imagine a situation where thread1 does zfs_mount() -> zfsvfs_setup() and its execution flow is right before the zil_open() call.
At the this point thread2 does zfs_ioc_rollback() -> zfs_suspend_fs() -> zfsvfs_teardown() and that sees a NULL value of z_log and does nothing.
Then, thread1 proceeds to zil_open() and thread2 performs the rollback and proceeds to zfs_resume_fs() -> zfsvfs_setup() of its own.
This is where the above panic can occur.

History

#1

Updated by Andriy Gapon about 3 years ago

I think that moving dmu_objset_set_user(zfsvfs->z_os, zfsvfs) call to the very end of zfsvfs_setup() should fix this issue.

#2

Updated by Electric Monk almost 3 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 90f2c094b3822f4825f21cef2c2faf7d03b55139

commit  90f2c094b3822f4825f21cef2c2faf7d03b55139
Author: Andriy Gapon <andriy.gapon@clusterhq.com>
Date:   2016-11-22T02:05:17.000Z

    7181 race between zfs_mount and zfs_ioc_rollback
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Also available in: Atom PDF