illumos gate

I've been doing lots of tests with other parts.

• Dell servers : because an old HP server does the same
• LSI 9207-8e or 8i which is based on 2308 chips : 2008 based controllers also make OmniOS crash
• A deep SAS topology : keeping a singler layer of expander doesn't help

• using any 12th generation Dell server we have : R320, R520, R720xd
• using a previous generation Dell R510 : unlike I said it does crash, test was incomplete
• using an old HP server with the same LSI 9207-8e HBAs
• using a Dell H200 IT mode controller (LSI 2008) both in R720xd's or the HP server
• updating enclosure expanders firmware to v. 55.14.18.0
• downgrading ST4000NM0023 drives to firmware 002
• using SAS switches in between HBAs and enclosures (LSI SAS6160)
• killing internal chaining in enclosures, by pluggin all expanders directly to HBAs or SAS switch

What I can't do is use an other brand/model of enclosures. Switching to other drives would also be complicated.

New crash dump with the R510 and simpler setup (one HBA, one path to one SAS switch) :
ftp://ftp.zeroloop.net/bugs/illumos/unix.28.xz
ftp://ftp.zeroloop.net/bugs/illumos/vmcore.28.xz
ftp://ftp.zeroloop.net/bugs/illumos/vmdump.28.xz

Is anyone able to troubleshoot with the files ? Would you need a remote SSH + KVM access ?
As far as I can tell, in all situations it's always the same bug where a lun struct gets a NULL vlun member.

Thanks

Since I got switches I'm covering more cases and here's an interesting one : with two enclosures (45 slots and 4 expanders each) but a single ST4000NM0023 drive it crashes !

Doing all this with Orcale Solaris 11 works, and I couldn't experience problems with Linux and Windows. Just saying this to confirm it can perform well with other systems and that's not a misuse of hardware or cable problem.

I wish to "+1" this bug report.

We have a Dell PowerEdge R710 server with 3 Dell MD1200 disk trays with 2TB SATA drives (12 drives per tray) and an LSI 9200-8e controller plus the internal controller from Dell (I believe it is a Dell H310). We are experiencing the same problem described by Thibault VINCENT with periodic freezes/lockups. We can trigger a crash with "mpathadm list lu" .

In addition, we have weird behavior with two Toshiba 2TB disks in the system, when the other 34 Seagate disks respond normally. Here is an excerpt from diskmap.py:

0:07:05 c10t5000C5001049C12Bd0 ST32000444SS 2.0T Ready (RDY) engr-std3: raidz1-5
0:07:06 c11t5000039428C8E13Bd0 MK2001TRKB 2.0T Ready (RDY) engr-std3: raidz1-6
0:07:06 c11t5000039428C8E13Bd0 MK2001TRKB 2.0T Ready (RDY) engr-std3: raidz1-6
0:07:07 c10t5000C500104991DFd0 ST32000444SS 2.0T Ready (RDY) engr-std3: raidz1-6

Notice the two Toshiba disks (MK2001TRKB) show in the same position in the MD1200 (enclosure 7, slot 6).

Here is an excerpt of format:

6. c9t5000039428C8E13Ad0 <TOSHIBA-MK2001TRKB-DCA8-1.82TB>
          /pci@0,0/pci8086,340e@7/pci1000,30b0@0/iport@f/disk@w5000039428c8e13a,0

41. c11t5000039428C8E13Bd0 <TOSHIBA-MK2001TRKB-DCA8-1.82TB>
          /pci@0,0/pci8086,340e@7/pci1000,30b0@0/iport@f0/disk@w5000039428c8e13b,0

that shows the two Toshiba disks. We also continually get reports of

genunix: [ID 408114 kern.info] /scsi_vhci/disk@g5000c50010468ba7 (sd58) offline

For a disk that as far as I can tell, never existed in the system (we have had a g5000c50010468ba6 in the system, but we removed and detached it, but it did not come back properly).

We're going to reinstall and try oi_151a8 but some progress on this bug would be appreciated. We will attach dumps once we have them.

Hi Frank,
I hope you'll get some attention with this bug, that was a show stopper for us and the reason why we moved away from Illumos based distros.
Nevertheless if some progress is made and depending how quick it comes, maybe I'll be able to confirm a fix on the initial Supermicro setup. It'll be available for lab for about two more weeks, then will be reduced to one JBOD and only few disks. Remote access is still an option for a developer to test kernels.

I can confirm this is a problem when using Supermicro JBODs, in particular the 847E26-RJBOD1.

The problem does not occur on my systems with DataON JBODs with the same OmniOS version.

I have this problem on a DR system that has many hours a day that I can run tests on. If any developer would like to utilize the this resource for diagnosis/testing please contact me.

It is possible this is a duplicate of #4085, and of this fix in illumos-nexenta:

commit 7cf584fc964e0765e7e5caa3db56c13d249741ca
Author: Jeffry Molanus <jeffry.molanus@nexenta.com>
Date: Tue May 13 05:49:12 2014 +0200

OS-253 we should not free mdi_pathinfo_t in mptsas when device(s) are retired

If anyone can reproduce this issue on OmniOS, please contact me.

Here's the analysis from #4085:

This has the same symptoms as an issue with retired devices with multiple paths we have identified and fixed, with the following explanation:
For each path in a multipath configuration, we have a pathinfo_t. It's
created in MDI_PATHINFO_STATE_INIT state (a "transient" state). For
the rest of its lifetime it's supposed to transition between either
OFFLINE or ONLINE depending on whether it is available.

In MDI, each pathinfo_t has a "client", which is the target device's
devinfo node. In mpt_sas's case, mptsas_create_virt_lun allocates a
pathinfo_t, and then tries to online the path via mdi_pi_online.

mdi_pi_online calls i_mdi_pi_state_change requesting
MDI_PATHINFO_STATE_ONLINE. This invokes the pHCI and vHCI specific
callbacks to set up the path. If it has a client and the driver is not
attached, it tries to invoke deferred attach via ndi_devi_online.
However, if ndi_devi_online fails, i_mdi_pi_state_change resets the
state of the pathinfo_t and also returns an error.

This is wrong because:
1) We have already invoked the callbacks to bring the path online.
Changing the path state flag to e.g. MDI_PATHINFO_STATE_OFFLINE (or
worse, back to MDI_PATHINFO_STATE_INIT) does not reflect the actual
state from the perspective of the pHCI/vHCI drivers.
2) Whether the driver could attach to the client should not affect
the availability of the path.

If the client device is retired, ndi_devi_online will fail, thus
sending us into a spiral of suffering.