Project

General

Profile

Bug #5183

illumos kernel panic and reboot on `zpool import` with a single disk zpool created on same system.

Added by Jeff Goeke-Smith over 5 years ago. Updated over 5 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
2014-09-18
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

Summary: A single disk zpool created on a SmartOS machine now causes a kernel crash dump and reboot when it is scanned by zpool import. The zpool was created on the same system that is now panicing. If I leave the drive plugged into the computer, this results in a loop of rebooting and crashing. Hot adding the drive to the machine results in the same crash when I run `zpool import` to scan for new pools.

Sequence that lead me here:

Boot and run SmartOS.
Create a single disk zpool as a temp space.
Copy ~1T of data to the temp drive.
Shutdown, and move the drive to a different port and a different controller.
Restart SmartOS. Machine crashes and reboots repeatedly until I remove the drive from the system.
Subsequent testing results in above summary.

I am running Joyent SmartOS version joyent_20140904T175324Z.
I have a LSI controler on the motherboard, in addition to the SATA ports. Both result in the same crash as far as I can
tell.
I can reproduce this failure. I have crash data from multiple runs. If necessary, I can provide remote access to this computer.

I'm pasting the content of ::msgbuf and attaching crash.0 as suggested at http://wiki.illumos.org/display/illumos/How+To+Report+Problems :

Thanks.
------------------------------------------------------------------------------------------------------------------------------
MESSAGE
nsmb0 is /pseudo/nsmb@0
/pci@0,0/pci8086,c05@1,1/pci15d9,691@0 (mpt_sas0):
mpt0: IOC Operational.
Universal TUN/TAP device driver ver 1.3.0 09/04/2014 (C) 1999-2000 Maxim Krasnyansky
pseudo-device: tap0
tap0 is /pseudo/tap@0
Universal TUN/TAP device driver ver 1.3.0 09/04/2014 (C) 1999-2000 Maxim Krasnyansky
pseudo-device: tun0
tun0 is /pseudo/tun@0
SMBIOS type 0x1, addr 0xca2
device rev. 1, firmware rev. 1.42, version 2.0
number of channels 2
watchdog supported
pseudo-device: lx_systrace0
lx_systrace0 is /pseudo/lx_systrace@0
/pci@0,0/pci8086,c05@1,1/pci15d9,691@0 (mpt_sas0):
mpt0: Initiator WWNs: 0x5003048011c07f00-0x5003048011c07f07
PCIE-device: pci15d9,691@0, mpt_sas0
PCI Express-device: pci15d9,691@0, mpt_sas0
mpt_sas0 is /pci@0,0/pci8086,c05@1,1/pci15d9,691@0
mpt_sas1 at mpt_sas0: scsi-iport 1
mpt_sas1 is /pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@1
/pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@1 (mpt_sas1) online
sd1 at scsi_vhci0: unit-address g50014ee20a444488: f_sym
sd1 is /scsi_vhci/disk@g50014ee20a444488
/scsi_vhci/disk@g50014ee20a444488 (sd1) online
/scsi_vhci/disk@g50014ee20a444488 (sd1) multipath status: degraded: path 1 mpt_sas1/disk@w50014ee20a444488,0 is online
mpt_sas2 at mpt_sas0: scsi-iport 2
mpt_sas2 is /pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@2
/pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@2 (mpt_sas2) online
sd2 at scsi_vhci0: unit-address g50014ee25f9c6d77: f_sym
sd2 is /scsi_vhci/disk@g50014ee25f9c6d77
/scsi_vhci/disk@g50014ee25f9c6d77 (sd2) online
/scsi_vhci/disk@g50014ee25f9c6d77 (sd2) multipath status: degraded: path 2 mpt_sas2/disk@w50014ee25f9c6d77,0 is online
mpt_sas3 at mpt_sas0: scsi-iport 4
mpt_sas3 is /pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@4
/pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@4 (mpt_sas3) online
sd3 at scsi_vhci0: unit-address g50014ee2098b79ac: f_sym
sd3 is /scsi_vhci/disk@g50014ee2098b79ac
/scsi_vhci/disk@g50014ee2098b79ac (sd3) online
/scsi_vhci/disk@g50014ee2098b79ac (sd3) multipath status: degraded: path 3 mpt_sas3/disk@w50014ee2098b79ac,0 is online
mpt_sas4 at mpt_sas0: scsi-iport 8
mpt_sas4 is /pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@8
/pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@8 (mpt_sas4) online
USB 1.10 device (usb4b3,3019) operating at low speed (USB 1.x) on USB 1.10 external hub: device@3, usb_mid1 at bus address 5
Lite-On Tech IBM USB Travel Keyboard with UltraNav
usb_mid1 is /pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@3
/pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@3 (usb_mid1) online
USB 1.10 device (usb6cb,9) operating at low speed (USB 1.x) on USB 1.10 external hub: device@4, usb_mid2 at bus address 6
Synaptics Inc. Composite TouchPad / TrackPoint
usb_mid2 is /pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@4
/pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@4 (usb_mid2) online
NOTICE: ahci0: hba AHCI version = 1.30
USB 1.10 interface (usbif4b3,3019.config1.0) operating at low speed (USB 1.x) on USB 1.10 external hub: keyboard@0, hid2 at bus address 5
Lite-On Tech IBM USB Travel Keyboard with UltraNav
hid2 is /pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@3/keyboard@0
/pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@3/keyboard@0 (hid2) online
USB 1.10 interface (usbif6cb,9.config1.0) operating at low speed (USB 1.x) on USB 1.10 external hub: mouse@0, hid4 at bus address 6
USB 1.10 interface (usbif4b3,3019.config1.1) operating at low speed (USB 1.x) on USB 1.10 external hub: input@1, hid3 at bus address 5
Synaptics Inc. Composite TouchPad / TrackPoint
Lite-On Tech IBM USB Travel Keyboard with UltraNav
hid4 is /pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@4/mouse@0
hid3 is /pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@3/input@1
/pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@3/input@1 (hid3) online
/pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@4/mouse@0 (hid4) online
USB 1.10 interface (usbif6cb,9.config1.1) operating at low speed (USB 1.x) on USB 1.10 external hub: mouse@1, hid5 at bus address 6
Synaptics Inc. Composite TouchPad / TrackPoint
hid5 is /pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@4/mouse@1
/pci@0,0/pci15d9,804@1d/hub@1/hub@5/device@4/mouse@1 (hid5) online
sd4 at scsi_vhci0: unit-address g5000c50065289d0b: f_sym
sd4 is /scsi_vhci/disk@g5000c50065289d0b
/scsi_vhci/disk@g5000c50065289d0b (sd4) online
/scsi_vhci/disk@g5000c50065289d0b (sd4) multipath status: degraded: path 4 mpt_sas4/disk@w5000c50065289d0b,0 is online
mpt_sas5 at mpt_sas0: scsi-iport 80
mpt_sas5 is /pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@80
/pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@80 (mpt_sas5) online
sd5 at scsi_vhci0: unit-address g50014ee2b4ef520e: f_sym
sd5 is /scsi_vhci/disk@g50014ee2b4ef520e
/scsi_vhci/disk@g50014ee2b4ef520e (sd5) online
/scsi_vhci/disk@g50014ee2b4ef520e (sd5) multipath status: degraded: path 5 mpt_sas5/disk@w50014ee2b4ef520e,0 is online
mpt_sas6 at mpt_sas0: scsi-iport v0
mpt_sas6 is /pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@v0
/pci@0,0/pci8086,c05@1,1/pci15d9,691@0/iport@v0 (mpt_sas6) online
ISA-device: asy0
asy0 is /pci@0,0/isa@1f/asy@1,3f8
ISA-device: asy1
asy1 is /pci@0,0/isa@1f/asy@1,2f8
ISA-device: asy2
asy2 is /pci@0,0/isa@1f/asy@1,3e8
ISA-device: pit_beep0
pit_beep0 is /pci@0,0/isa@1f/pit_beep
NOTICE: igb0 registered
NOTICE: igb1 registered
NOTICE: igb0: igb 2.3.8-ish
NOTICE: igb1: igb 2.3.8-ish
dump on /dev/zvol/dsk/zones/dump size 4096 MB
device pciclass,030000@0(display#0) keeps up device sd@0,0(disk#0), but the former is not power managed
NOTICE: igb1 link up, 1000 Mbps, full duplex
Creating /etc/devices/devid_cache
Creating /etc/devices/pci_unitaddr_persistent
WARNING: /pci@0,0/pci15d9,804@1f,2:
SATA device detected at port 0
/pci@0,0/pci15d9,804@1f,2 :
SATA disk device at port 0
model ST32000542AS
firmware CC34
serial number 5XW1QEWT
supported features:
48-bit LBA, DMA, Native Command Queueing, SMART, SMART self-test
SATA Gen2 signaling speed (3.0Gbps)
Supported queue depth 32
capacity = 3907029168 sectors
sd6 at ahci0: target 0 lun 0
sd6 is /pci@0,0/pci15d9,804@1f,2/disk@0,0
/pci@0,0/pci15d9,804@1f,2/disk@0,0 (sd6) online

panic[cpu4]/thread=ffffff04e8f34ae0:
assertion failed: zap_lookup(ddt->ddt_os, ddt->ddt_spa->spa_ddt_stat_object, name, sizeof (uint64_t), sizeof (ddt_histogram_t) / sizeof (uint
64_t), &ddt->ddt_histogram[type][class]) 0 (0x32 0x0), file: ../../common/fs/zfs/ddt.c, line: 119

ffffff001fc57830 fffffffffba6740d ()
ffffff001fc57930 zfs:ddt_object_load+192 ()
ffffff001fc57990 zfs:ddt_load+cd ()
ffffff001fc57a80 zfs:spa_load_impl+a51 ()
ffffff001fc57b20 zfs:spa_load+14e ()
ffffff001fc57b70 zfs:spa_tryimport+aa ()
ffffff001fc57bc0 zfs:zfs_ioc_pool_tryimport+51 ()
ffffff001fc57c70 zfs:zfsdev_ioctl+4a7 ()
ffffff001fc57cb0 genunix:cdev_ioctl+39 ()
ffffff001fc57d00 specfs:spec_ioctl+60 ()
ffffff001fc57d90 genunix:fop_ioctl+55 ()
ffffff001fc57eb0 genunix:ioctl+9b ()
ffffff001fc57f10 unix:brand_sys_sysenter+1d3 ()

syncing file systems...
done
dumping to /dev/zvol/dsk/zones/dump, offset 65536, content: kernel



Files

crash.0 (544 KB) crash.0 Jeff Goeke-Smith, 2014-09-18 03:57 PM

History

#1

Updated by Jeff Goeke-Smith over 5 years ago

Additional information. Using a usb sata adapter, and a zpool import with the -F and readonly options, I was able to import the drive/pool enough to copy my data off the device.

Post data transfer, I attempted my test again. The drive still causes a kernel panic on `zpool import`.

If I don't see any interest in this bug or gathering more debugging data in the next few days, I'm going to zeroize the drive, and put it back into production. When that happens, I can't recreate the failure anymore.

Also available in: Atom PDF