illumos gate

The original bug report at Delphix:

All the attempts to mount iscsi luns on
the target server are failing. iscsi initiator on the target server is in
"reconnecting" state for the Delphix target. Manual attempts to connect to the
Delphix iscsi target port also failed from the target host. netstat on Delphix
server did not show any connection listening on port 3260(iscsi target port).
Rebooted(reboot -d) the Delphix server to get out of this state.

> ::stacks -m idm
THREAD           STATE    SOBJ                COUNT
ffffff007c3fbc40 SLEEP    CV                      1
                 swtch+0x145
                 cv_timedwait_hires+0xe0
                 cv_reltimedwait+0x4f
                 idm_wd_thread+0x1ec
                 thread_start+8

ffffff007c3f5c40 SLEEP    CV                      1
                 swtch+0x145
                 cv_wait+0x61
                 idm_refcnt_wait_ref+0x7d
                 iscsit_conn_destroy+0x5e
                 iscsit_client_notify+0x3d
                 idm_notify_client+0x6b
                 idm_conn_unref+0x3e
                 idm_refcnt_unref_task+0x5e
                 taskq_thread+0x285
                 thread_start+8

The listener thread(idm_so_svc_port_watcher) is missing which explains why
there is no connection listening on the port 3260.

Jeff Biseda's analysis:

In idm_so_port_watcher() we have:

1242    while (so_svc->is_thread_running) {
1243        mutex_exit(&svc->is_mutex);
1244
1245        if ((rc = ksocket_accept(so_svc->is_so,
1246            (struct sockaddr *)&t_addr, &t_addrlen,
1247            &new_so, CRED())) != 0) {
1248            mutex_enter(&svc->is_mutex);
1249            if (rc == ECONNABORTED)
1250                continue;
1251            /* Connection problem */
1252            break;

If we get some error back other than ECONNABORTED the watcher thread will exit.

After running a patch on the customer system we confirmed that this was failing
with EINTR.

I decided to keep the thread running until is_thread_running is set to false in
an orderly shutdown. We saw that when this thread exited unexpectedly idm got
into a strange state when the target removal was waiting for a cv that wouldn't
be signaled.