Project

General

Profile

Bug #4085

mpathadm list lu cause panic

Added by Zhiwen Zheng about 6 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
driver - device drivers
Start date:
2013-08-28
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

config:

lsi sas2008
lsi 6G expander
6 bad disks (I am testing error handling of the storage subsystem)

fmadm faulty detects 3
format can list 2
sasinfo lu list 2
cfgadm -al sometimes list 2, sometimes list 3 (why ?)

::msgbuf
MESSAGE                                                               
         es:               4b  fs:                0  gs:              1c3
        trp:                d err:                0 rip: fffffffffbe8c348
         cs:               30 rfl:            10286 rsp: ffffff000fd31a60
         ss:               38

cpu          address    timestamp type  vc  handler   pc
  0 fffffffffbc1de10  21de56a652d 
trap   d 
     #gp 
ddi_get_instance+8
  0 fffffffffbc1dc98  21de56a3559 
sysc  36 
   ioctl 
feef8a55
  0 fffffffffbc1db20  21de569f029 
sysc   6 
   close 
feef8735
  0 fffffffffbc1d9a8  21de569e4a9 
intr  ff 
unknown  
fakesoftint+23
  0 fffffffffbc1d830  21de569da35 
intr  ff 
unknown  
fakesoftint+23
  0 fffffffffbc1d6b8  21de569c8b9 
sysc  c9 
doorfs32 
feef9a8e
  0 fffffffffbc1d540  21de569bbc9     
intr  ff 
unknown  
fakesoftint+23
  0 fffffffffbc1d3c8  21de569ac01 
intr  ff 
unknown  
fakesoftint+23
  0 fffffffffbc1d250  21de5699141 
intr  ff 
unknown  
fakesoftint+23
  0 fffffffffbc1d0d8  21de569842d 
sysc  c9 
doorfs32 
feef9848
  1 ffffff02edf4f780  21de56a32e1 
intr  ff 
unknown  
fakesoftint+23
  1 ffffff02edf4f608  21de56a254d 
intr  ff 
unknown  
fakesoftint+23
  1 ffffff02edf4f490  21de56a1fad 
intr  ff 
unknown  
fakesoftint+23
  1 ffffff02edf4f318  21de569f389 
sysc  b7 
 pollsys 
feef8ce5                              
  1 ffffff02edf4f1a0  21de569e7ad 
intr  ff 
unknown  
fakesoftint+23
  1 ffffff02edf4f028  21de569d5d9 
sysc  4d 
syslwp_park 
feef4649
  1 ffffff02edf4eeb0  21de5698469 
sysc  55 
getmsg32 
feef8945
  1 ffffff02edf4ed38  21de56975bd 
intr  ff 
unknown  
fakesoftint+23
  1 ffffff02edf4ebc0  21de5694fbd 
intr  ff 
unknown  
fakesoftint+23
  1 ffffff02edf4ea48  21de5641ad9 
intr  ff 
unknown  
fakesoftint+23
  2 ffffff02ee136cd8  21de56a6b1d 
intr  ff 
unknown  
fakesoftint+23
  2 ffffff02ee136b60  21de56a598d 
sysc  4d 
syslwp_park                           
feef4619
  2 ffffff02ee1369e8  21de56a2281 
intr  ff 
unknown  
fakesoftint+23
  2 ffffff02ee136870  21de56a0f61 
intr  ff 
unknown  
fakesoftint+23
  2 ffffff02ee1366f8  21de56176ef 
intr  f0 
 xc_serv 
i86_mwait+d
  2 ffffff02ee136580  21de542126d 
intr  f0 
 xc_serv 
i86_mwait+d
  2 ffffff02ee136408  21de53a4d6a 
intr  f0 
 xc_serv 
i86_mwait+d
  2 ffffff02ee136290  21de5377e02 
intr  f0 
 xc_serv 
i86_mwait+d
  2 ffffff02ee136118  21de536dc46 
intr  ff 
unknown  
fakesoftint+23
  2 ffffff02ee135fa0  21de536d48a 
intr  ff                              
unknown  
fakesoftint+23
  3 ffffff02ee212190  21de5696279 
intr  ff 
unknown  
fakesoftint+23
  3 ffffff02ee212018  21de5695809 
intr  ff 
unknown  
fakesoftint+23
  3 ffffff02ee211ea0  21de569523d 
intr  ff 
unknown  
fakesoftint+23
  3 ffffff02ee211d28  21de5693abd 
intr  ff 
unknown  
fakesoftint+23
  3 ffffff02ee211bb0  21de5690de9 
intr  ff 
unknown  
fakesoftint+23
  3 ffffff02ee211a38  21de5617472 
intr  f0 
 xc_serv 
i86_mwait+d
  3 ffffff02ee2118c0  21de55f0bca 
intr  18 
ahci_intr 
i86_mwait+d
  3 ffffff02ee211748  21de5421502     
intr  f0 
 xc_serv 
i86_mwait+d
  3 ffffff02ee2115d0  21de53a4d94 
intr  f0 
 xc_serv 
disp_getwork+1
  3 ffffff02ee211458  21de53838e2 
intr  18 
ahci_intr 
i86_mwait+d

ffffff000fd31850 unix:real_mode_stop_cpu_stage2_end+a683 ()
ffffff000fd31960 unix:trap+bfb ()
ffffff000fd31970 unix:cmntrap+1ca ()
ffffff000fd31a70 genunix:ddi_get_instance+8 ()
ffffff000fd31ac0 scsi_vhci:vhci_mpapi_sync_lu_oid_list+5d ()
ffffff000fd31b70 scsi_vhci:vhci_mpapi_ioctl+b1 ()
ffffff000fd31c30 scsi_vhci:vhci_mpapi_ctl+113 ()
ffffff000fd31c70 scsi_vhci:vhci_ioctl+5d ()
ffffff000fd31cb0 genunix:cdev_ioctl+39 ()
ffffff000fd31d00 specfs:spec_ioctl+60 ()
ffffff000fd31d90 genunix:fop_ioctl+55 ()
ffffff000fd31eb0 genunix:ioctl+9b ()
ffffff000fd31f00 unix:brand_sys_sysenter+2b7 ()

syncing file systems...
 done
dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
NOTICE: ahci0: ahci_tran_reset_dport port 0 reset port

root@spa:~# cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c6                             scsi-sas     connected    configured   unknown
c6::es/ses0                    ESI          connected    configured   unknown
c6::smp/expd0                  smp          connected    configured   unknown
c6::w500605ba009c09c9,0        disk-path    connected    configured   unknown
c6::w500605ba009c11ed,0        disk-path    connected    configured   unknown
c6::w500605ba009c0909,0        disk-path    connected    configured   unknown
c7                             scsi-sas     connected    unconfigured unknown
sata0/0::dsk/c5t0d0            disk         connected    configured   ok
usb8/1                         usb-hub      connected    configured   ok
usb8/1.1                       unknown      empty        unconfigured ok
usb8/1.2                       unknown      empty        unconfigured ok
usb8/1.3                       unknown      empty        unconfigured ok
usb8/1.4                       unknown      empty        unconfigured ok
usb8/1.5                       unknown      empty        unconfigured ok
usb8/1.6                       unknown      empty        unconfigured ok
usb8/2                         unknown      empty        unconfigured ok
usb9/1                         usb-hub      connected    configured   ok
usb9/1.1                       unknown      empty        unconfigured ok
usb9/1.2                       unknown      empty        unconfigured ok
usb9/1.3                       unknown      empty        unconfigured ok
usb9/1.4                       unknown      empty        unconfigured ok
usb9/1.5                       unknown      empty        unconfigured ok
usb9/1.6                       unknown      empty        unconfigured ok
usb9/2                         unknown      empty        unconfigured ok
cfgadm: Configuration administration not supported: Error: hotplug service is probably not running, please use 'svcadm enable hotplug' to enable the service. See cfgadm_shp(1M) for more details.
root@spa:~# sasinfo lu
OS Device Name: /dev/es/ses0
OS Device Name: /dev/rdsk/c0t5000C5001A63446Ed0s2
OS Device Name: /dev/rdsk/c0t5000C5001A5502C7d0s2
root@spa:~# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c0t5000C5001A5502C7d0 <ATA-ST31000340NS-SN06-931.51GB>
          /scsi_vhci/disk@g5000c5001a5502c7
       1. c0t5000C5001A63446Ed0 <ATA-ST31000340NS-SN06-931.51GB>
          /scsi_vhci/disk@g5000c5001a63446e
       2. c5t0d0 <ATA-8GBSATAFlashD-A01A cyl 3817 alt 2 hd 128 sec 32>
          /pci@0,0/pci8086,3b20@1f,2/disk@0,0

fmadm faulty
--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Aug 28 17:19:13 7cf9354b-8aa3-c30f-b4e7-8dbeec996460  DISK-8000-3E   Critical  

Host        : spa
Platform    : To-be-filled-by-O.E.M.    Chassis_id  : To-be-filled-by-O.E.M.
Product_sn  : 

Fault class : fault.io.scsi.cmd.disk.dev.rqs.derr
Affects     : dev:///:devid=id1,sd@n5000c5001a54a2ee//scsi_vhci/disk@g5000c5001a54a2ee
                  faulted and taken out of service
FRU         : "SLOT 06" (hc://:product-id=Qsan-J300H:server-id=:chassis-id=5001378005004000:serial=9QJ6FXJS:part=ATA-ST31000340NS:revision=SN06/ses-enclosure=0/bay=6/disk=0)
                  faulty

Description : A non-recoverable hardware failure was detected by the device
              while performing a command.
              Refer to http://illumos.org/msg/DISK-8000-3E for more
              information.

Response    : The device may be offlined or degraded.

Impact      : The device has failed. The service may have been lost or
              degraded.

Action      : Ensure that the latest drivers and patches are installed.
              Schedule a repair procedure to replace the affected
              device. Use 'fmadm faulty' to find the affected disk.

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Aug 28 17:31:58 5e7d4fcc-15ba-c636-adcb-dbe028b7d87f  SUNOS-8000-KL  Major     

Host        : spa
Platform    : To-be-filled-by-O.E.M.    Chassis_id  : To-be-filled-by-O.E.M.
Product_sn  : 

Fault class : defect.sunos.kernel.panic
Affects     : sw:///:path=/var/crash/unknown/.5e7d4fcc-15ba-c636-adcb-dbe028b7d87f
                  faulted but still in service
Problem in  : sw:///:path=/var/crash/unknown/.5e7d4fcc-15ba-c636-adcb-dbe028b7d87f
                  faulted but still in service

Description : The system has rebooted after a kernel panic.  Refer to
              http://illumos.org/msg/SUNOS-8000-KL for more information.

Response    : The failed system image was dumped to the dump device.  If
              savecore is enabled (see dumpadm(1M)) a copy of the dump will be
              written to the savecore directory /var/crash/unknown.

Impact      : There may be some performance impact while the panic is copied to
              the savecore directory.  Disk space usage by panics can be
              substantial.

Action      : If savecore is not enabled then please take steps to preserve the
              crash image.
              Use 'fmdump -Vp -u 5e7d4fcc-15ba-c636-adcb-dbe028b7d87f' to view
              more panic detail.  Please refer to the knowledge article for
              additional information.

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Aug 28 17:18:02 7e0717ed-1fb1-e602-d432-c4494ab413ce  DISK-8000-0X   Major     

Host        : spa
Platform    : To-be-filled-by-O.E.M.    Chassis_id  : To-be-filled-by-O.E.M.
Product_sn  : 

Fault class : fault.io.disk.predictive-failure
Affects     : dev:///:devid=id1,sd@n5000c5001a69e6f6//scsi_vhci/disk@g5000c5001a69e6f6
                  faulted and taken out of service
FRU         : "SLOT 01" (hc://:product-id=Qsan-J300H:server-id=:chassis-id=5001378005004000:serial=9QJ6PXY7:part=ATA-ST31000340NS:revision=SN06/ses-enclosure=0/bay=1/disk=0)
                  faulty

Description : SMART health-monitoring firmware reported that a disk
              failure is imminent.
              Refer to http://illumos.org/msg/DISK-8000-0X for more
              information.

Response    : None.

Impact      : It is likely that the continued operation of
              this disk will result in data loss.

Action      : Schedule a repair procedure to replace the affected disk.
              Use fmdump -v -u <EVENT_ID> to identify the disk.

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Aug 28 17:44:33 bbfbfdfa-fa81-61fb-8de6-cc42edf1e68b  SUNOS-8000-KL  Major     

Host        : spa
Platform    : To-be-filled-by-O.E.M.    Chassis_id  : To-be-filled-by-O.E.M.
Product_sn  : 

Fault class : defect.sunos.kernel.panic
Affects     : sw:///:path=/var/crash/unknown/.bbfbfdfa-fa81-61fb-8de6-cc42edf1e68b
                  faulted but still in service
Problem in  : sw:///:path=/var/crash/unknown/.bbfbfdfa-fa81-61fb-8de6-cc42edf1e68b
                  faulted but still in service

Description : The system has rebooted after a kernel panic.  Refer to
              http://illumos.org/msg/SUNOS-8000-KL for more information.

Response    : The failed system image was dumped to the dump device.  If
              savecore is enabled (see dumpadm(1M)) a copy of the dump will be
              written to the savecore directory /var/crash/unknown.

Impact      : There may be some performance impact while the panic is copied to
              the savecore directory.  Disk space usage by panics can be
              substantial.

Action      : If savecore is not enabled then please take steps to preserve the
              crash image.
              Use 'fmdump -Vp -u bbfbfdfa-fa81-61fb-8de6-cc42edf1e68b' to view
              more panic detail.  Please refer to the knowledge article for
              additional information.

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Aug 28 17:16:04 d4eaee12-cf12-e53f-bc9b-b7541c96028c  DISK-8000-3E   Critical  

Host        : spa
Platform    : To-be-filled-by-O.E.M.    Chassis_id  : To-be-filled-by-O.E.M.
Product_sn  : 

Fault class : fault.io.scsi.cmd.disk.dev.rqs.derr
Affects     : dev:///:devid=id1,sd@n5000c5004d449af7//scsi_vhci/disk@g5000c5004d449af7
                  faulted and taken out of service
FRU         : "SLOT 05" (hc://:product-id=Qsan-J300H:server-id=:chassis-id=5001378005004000:serial=Z292VP8A:part=ATA-ST32000645NS:revision=0004/ses-enclosure=0/bay=5/disk=0)
                  faulty

Description : A non-recoverable hardware failure was detected by the device
              while performing a command.
              Refer to http://illumos.org/msg/DISK-8000-3E for more
              information.

Response    : The device may be offlined or degraded.

Impact      : The device has failed. The service may have been lost or
              degraded.

Action      : Ensure that the latest drivers and patches are installed.
              Schedule a repair procedure to replace the affected
              device. Use 'fmadm faulty' to find the affected disk.

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Aug 28 17:36:51 dfd4e570-457c-6bcb-ece6-89b83a91f89f  SUNOS-8000-KL  Major     

Host        : spa
Platform    : To-be-filled-by-O.E.M.    Chassis_id  : To-be-filled-by-O.E.M.
Product_sn  : 

Fault class : defect.sunos.kernel.panic
Affects     : sw:///:path=/var/crash/unknown/.dfd4e570-457c-6bcb-ece6-89b83a91f89f
                  faulted but still in service
Problem in  : sw:///:path=/var/crash/unknown/.dfd4e570-457c-6bcb-ece6-89b83a91f89f
                  faulted but still in service

Description : The system has rebooted after a kernel panic.  Refer to
              http://illumos.org/msg/SUNOS-8000-KL for more information.

Response    : The failed system image was dumped to the dump device.  If
              savecore is enabled (see dumpadm(1M)) a copy of the dump will be
              written to the savecore directory /var/crash/unknown.

Impact      : There may be some performance impact while the panic is copied to
              the savecore directory.  Disk space usage by panics can be
              substantial.

Action      : If savecore is not enabled then please take steps to preserve the
              crash image.
              Use 'fmdump -Vp -u dfd4e570-457c-6bcb-ece6-89b83a91f89f' to view
              more panic detail.  Please refer to the knowledge article for
              additional information.

Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:42 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:42 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:43 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:43 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:44 spa ipf: [ID 774698 kern.info] IP Filter: v4.1.9, running.
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:44 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:44 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:45 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:45 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:46 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:46 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a69e6f6 (sd17) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd8: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5004d449af7 (sd8) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd20: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a54a2ee (sd20) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd16: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c5001a3e6439 (sd16) offline
Aug 28 17:43:47 spa genunix: [ID 649217 kern.warning] WARNING: mdi: sd17: i_mdi_pi_state_change: ndi_devi_online failed error ffffffff
Aug 28 17:43:47 spa genunix: [ID 408114 kern.info] /

Related issues

Related to illumos gate - Bug #4051: Don't free mdi_pathinfo_t in mptsas when device(s) are retired.Closed2013-08-17

Actions

History

#1

Updated by Albert Lee over 5 years ago

  • Status changed from New to In Progress
  • Assignee set to Albert Lee
  • % Done changed from 0 to 90

This has the same symptoms as an issue with retired devices with multiple paths we have identified and fixed, with the following explanation:
For each path in a multipath configuration, we have a pathinfo_t. It's
created in MDI_PATHINFO_STATE_INIT state (a "transient" state). For
the rest of its lifetime it's supposed to transition between either
OFFLINE or ONLINE depending on whether it is available.

In MDI, each pathinfo_t has a "client", which is the target device's
devinfo node. In mpt_sas's case, mptsas_create_virt_lun allocates a
pathinfo_t, and then tries to online the path via mdi_pi_online.

mdi_pi_online calls i_mdi_pi_state_change requesting
MDI_PATHINFO_STATE_ONLINE. This invokes the pHCI and vHCI specific
callbacks to set up the path. If it has a client and the driver is not
attached, it tries to invoke deferred attach via ndi_devi_online.
However, if ndi_devi_online fails, i_mdi_pi_state_change resets the
state of the pathinfo_t and also returns an error.

This is wrong because:
1) We have already invoked the callbacks to bring the path online.
Changing the path state flag to e.g. MDI_PATHINFO_STATE_OFFLINE (or
worse, back to MDI_PATHINFO_STATE_INIT) does not reflect the actual
state from the perspective of the pHCI/vHCI drivers.
2) Whether the driver could attach to the client should not affect
the availability of the path.

If the client device is retired, ndi_devi_online will fail, thus
sending us into a spiral of suffering.

#2

Updated by Dan McDonald over 4 years ago

It is possible this is a duplicate of #4051, and of this fix in illumos-nexenta:

commit 7cf584fc964e0765e7e5caa3db56c13d249741ca
Author: Jeffry Molanus <>
Date: Tue May 13 05:49:12 2014 +0200

OS-253 we should not free mdi_pathinfo_t in mptsas when device(s) are retired

If you can reproduce this on OmniOS, please contact me.

#3

Updated by Dan McDonald over 4 years ago

  • Status changed from In Progress to Feedback
  • % Done changed from 90 to 100

Closing as a duplicate of #4051.

#4

Updated by Dan McDonald over 4 years ago

  • Status changed from Feedback to Closed

Also available in: Atom PDF