illumos gate

During continuous write on zpool disk I/O halts for couple of seconds. iostat reports 100% busy disk and hw error on different disks. system reports scsi timeout error message on console and syslog messages.

Server Model: Dell R730xd
SAS Controller: PERC H730 Mini

root@dell-r730xd:/root# uname -a
SunOS dell-r730xd 5.11 omnios-d08e0e5 i86pc i386 i86pc

root@dell-r730xd:/root# modinfo |grep mr_sas
180 fffffffff80d8000  1d070 139   1  mr_sas (6.503.00.00ILLUMOS)

root@dell-r730xd:/root# digest -a sha1 /kernel/drv/amd64/mr_sas
2c81e48297a06a585cf5ae31ee92a537e7563962

iostat -exn 1 output:
                            extended device statistics       ---- errors ---
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b s/w h/w trn tot device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 rpool
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c3t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c1t0d1
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c1t1d1
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c1t2d1
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c1t3d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  71  42 113 c1t4d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  51  28  79 c1t5d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  59  30  89 c1t6d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  75  47 122 c1t7d1
    0.0    0.0    0.0    0.0  0.0 10.0    0.0    0.0   0 100   0  57  27  84 c1t8d1
    0.0    0.0    0.0    0.0  0.0  9.0    0.0    0.0   0 100   0  99  63 162 c1t9d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  74  49 123 c1t10d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  76  40 116 c1t11d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  60  30  90 c1t12d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  86  59 145 c1t13d1
    0.0    0.0    0.0    0.0  8.0  1.0    0.0    0.0 100 100   0  71  40 111 c1t14d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0 118  73 191 c1t15d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  79  47 126 c1t16d1
    0.0    0.0    0.0    0.0  0.0  8.0    0.0    0.0   0 100   0  85  54 139 c1t17d1
    0.0    0.0    0.0    0.0  8.0  1.0    0.0    0.0 100 100   0  83  52 135 c1t18d1
    0.0    0.0    0.0    0.0  8.0  1.0    0.0    0.0 100 100   0  94  64 158 c1t19d1
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c1t20d1
    0.0    0.0    0.0    0.0 24715.0 134.0    0.0    0.0 100 100   0   0   0   0 test

/var/adm/messages:

Sep 15 12:16:13 dell-r730xd genunix: [ID 761523 kern.warning] WARNING: mr_sas0: tbolt_complete_cmd: scsi_io failed
Sep 15 12:21:59 dell-r730xd last message repeated 267 times
Sep 15 12:21:59 dell-r730xd genunix: [ID 761523 kern.warning] WARNING: mr_sas0: tbolt_complete_cmd: scsi_io failed

root@dell-r730xd:/root# megacli -AdpAllInfo -a0

Adapter #0

==============================================================================
                    Versions
                ================
Product Name    : PERC H730 Mini
Serial No       : 57I009V
FW Package Build: 25.3.0.0016

                    Mfg. Data
                ================
Mfg. Date       : 07/21/15
Rework Date     : 07/21/15
Revision No     : A01
Battery FRU     : N/A

                Image Versions in Flash:
                ================
BIOS Version       : 6.23.03.0_4.16.07.00_0x060C0200
Ctrl-R Version     : 5.04-0012
FW Version         : 4.250.00-4402
NVDATA Version     : 3.1411.00-0019
Boot Block Version : 3.02.00.00-0001

root@dell-r730xd:/root# megacli -AdpAllInfo -a0 | egrep 'JBOD|Direct PD'
Enable JBOD                             : Yes
Direct PD Mapping                : Yes
Enable JBOD                      : Yes