illumos gate

We had a system that would have the zfs deadman seemingly randomly fire on the systems boot zpool. Attempting to do a crash dump failed since I/O to the mirrored boot drives seemed to be hung. We were able to get kmdb loaded and capture the state of the system after this occurred. Unfortunately, the serial console was not configured, so there are only screenshots of the kmdb output (though I'm attempting to reproduce w/ the serial console configured so I can copy/paste the kmdb analysis and will update if possible).

Looking at threads in ahci (::stacks -m ahci), I saw three threads. One thread was servicing an ahci interrupt and had a call stack containing (I'm abbreviating this w/ the key bits since I'm having to copy it by hand):

mutex_vector_enter
sata`sata_scsi_init_pkt
scsi`scsi_init_pkt
...
scsi`scsi_hba_pkt_comp
sata`sata_txlt_rw_completion
ahci`flush_doneq
ahci`ahci_intr_ncq_events
ahci`ahci_intr_set_device_bits
ahci`ahci_port_intr
ahci`ahci_intr

The mutex in question is from this (in sata_scsi_init_pkt):

mutex_enter(&(SATA_CPORT_MUTEX(sata_hba_inst,
          sata_device.satadev_addr.cport)));

Examining who held the mutex, it was a thread with the following call stack (again abbreviated):

...
cv_wait
ahci`ahci_do_sync_start
ahci`ahci_tran_start
sata`sata_fetch_device_identify_data
sata`sata_txlt_inquiry
sata`sata_scsi_start
scsi`scsi_transport
sd`sd_start_cmds
sd`sd_restore_throttle
...
taskq_thread

The above mentioned mutex is acquired in sata_txlt_inquiry and is held for effectively the entire function. Note that the mutex associated with the blocked CV is not the cport_mutex. We are blocked due to the following code:

    while (spkt->satapkt_reason == SATA_PKT_BUSY)
              cv_wait(&ahci_portp->ahciport_cv,
                  &ahci_portp->ahciport_mutex);

So the command is waiting for completion with the same sata framework cport_mutex being held that the ISR is blocked on. Eventually, the ahci watchdog thread attempts to time out the request, it resets the port

mutex_vector_enter
sata`sata_hba_event_notify
ahci`ahci_restart_port_wait_till_ready
ahci`ahci_timeout_pkts
ahci`ahci_watchdog_handler

In this case the watchdog handler has reset the port, and is blocked on the sata framework on the same cport_mutex trying to notify the sata framework that the port has been reset, so it also cannot make progress and clean things up (i.e. complete the synchronous request as failed/timed out).

I believe the error is sata_txlt_inquiry holding cport_mutex while calling into the HBA (the sata_tran_start method i.e. ahci_tran_start). No other callers in the SATA framework do this -- either they just don't have the mutex held, or they explicitly drop it around the call into the HBA). We (unfortunately) don't have any sort of comments or docs to point at to definitely say it's wrong, but the evidence is highly suggestive that is the correct way to resolve the deadlock.