Project

General

Profile

Bug #11056

vdev_disk_io_start will panic the system if ldi_strategy returns an error.

Added by Jerry Jelinek 4 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
zfs - Zettabyte File System
Start date:
2019-05-21
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

We have hit a race condition a couple of times in production. Here is the dump info

> ::status
debugging crash dump vmcore.0 (64-bit) from JA814788
operating system: 5.11 joyent_20180913T001833Z (i86pc)
git branch: release-20180913
git rev: 76683f77aeee2619b447d8ea5843399835fbc586
image uuid: (not set)
panic message: assertion failed: ldi_strategy(dvd->vd_lh, bp) == 0, file: ../../common/fs/zfs/vdev_disk.c, line: 809
dump content: kernel pages only
> $C
fffffb041586c8a0 vpanic()
fffffb041586c8d0 0xfffffffffba9d2a8()
fffffb041586c910 vdev_disk_io_start+0x124(fffffdd79b70e3f0)
fffffb041586c970 zio_vdev_io_start+0xac(fffffdd79b70e3f0)
fffffb041586c9a0 zio_execute+0x7f(fffffdd79b70e3f0)
fffffb041586c9e0 vdev_queue_io_done+0x68(fffffeb72cf7a818)
fffffb041586ca20 zio_vdev_io_done+0x88(fffffeb72cf7a818)
fffffb041586ca50 zio_execute+0x7f(fffffeb72cf7a818)
fffffb041586cb10 taskq_thread+0x2d0(fffffdc3ebf752e8)
fffffb041586cb20 thread_start+8()
> fffffdd79b70e3f0::print zio_t io_vd
io_vd = 0xfffffdc3e679f800
> fffffdc3e679f800::print vdev_t vdev_tsd
vdev_tsd = 0xfffffdc3eba4c580
> fffffdc3eba4c580::print vdev_disk_t
{
    vd_devid = 0
    vd_minor = 0
    vd_lh = 0
    vd_ldi_cbs = {
        list_size = 0x18
        list_offset = 0
        list_head = {
            list_next = 0xfffffdc3eabbe910
            list_prev = 0xfffffdc3eb945060
        }
    }
    vd_ldi_offline = 0x1 (B_TRUE)
}

History

#1

Updated by Jerry Jelinek 4 months ago

In vdev_disk_io_start we check for an offline device and a NULL vd_lh at the beginning of the function, but nothing prevents this from changing after that.

#2

Updated by Electric Monk 4 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit fa88c70fa54c01be933cc49fd271e5c4468f7eb8

commit  fa88c70fa54c01be933cc49fd271e5c4468f7eb8
Author: Jerry Jelinek <jerry.jelinek@joyent.com>
Date:   2019-05-28T14:35:51.000Z

    11056 vdev_disk_io_start will panic the system if ldi_strategy returns an error.
    Reviewed by: Toomas Soome <tsoome@me.com>
    Reviewed by: Andy Stormont <astormont@racktopsystems.com>
    Reviewed by: Kody Kantor <kody.kantor@joyent.com>
    Approved by: Gordon Ross <gordon.ross@nexenta.com>

Also available in: Atom PDF