illumos gate

Analysis by Jeff Biseda of a panic we hit at Delphix:

> $C
ffffff007b4c69e0 vpanic()
ffffff007b4c6a10 mutex_panic+0x73(fffffffffb961bd0, ffffff120f9fa2e0)
ffffff007b4c6a70 mutex_vector_enter+0x446(ffffff120f9fa2e0)
ffffff007b4c6ac0 idm_conn_event+0x36(ffffff120f9f9000, 12, 0)
ffffff007b4c6ae0 idm_login_timeout+0x1d(ffffff120f9f9000)
ffffff007b4c6b20 callout_list_expire+0x77(ffffff113dda8fc0, ffffff11b6480100)
ffffff007b4c6b50 callout_expire+0x31(ffffff113dda8fc0)
ffffff007b4c6b70 callout_execute+0x1e(ffffff113dda8fc0)
ffffff007b4c6c20 taskq_thread+0x285(ffffff1159675490)
ffffff007b4c6c30 thread_start+8()

Is the idm_conn_t still around?

> ffffff120f9fa2e0::whatis
ffffff120f9fa2e0 is ffffff120f9f9000+12e0, freed from kmem_alloc_8192

No, it's been freed, and indeed, it's not in the global list.

> ::iscsi_conn
idm_conn_t       Type   Transport     State/FFP
ffffff120ba8f000 Tgt    SOCKETS         5/1     
ffffff135e250000 Tgt    SOCKETS         5/1     
ffffff115a05b000 Tgt    SOCKETS         5/1     
ffffff135fdc8000 Tgt    SOCKETS         5/1     
ffffff1359eba000 Tgt    SOCKETS         5/1  

We're being called back here from a routine queued by timeout().

I see numerous calls to untimeout() at various places in the state machine.
There must be some transition which is failing to call untimeout().

I parsed through the audit buffers and this is the history of the state machine
transitions that we went through for this connection:

CS_S1_FREE: CE_CONNECT_ACCEPT
SC_S1_FREE -> SC_S3_XPT_UP
SC_S3_XPT_UP: CE_LOGIN_RCV
SC_S3_XPT_UP -> CS_S4_IN_LOGIN
CS_S4_IN_LOGIN: CE_MISC_TX
CS_S4_IN_LOGIN: CE_LOGIN_RCV
CS_S4_IN_LOGIN: CE_MISC_TX
CS_S4_IN_LOGIN: CE_LOGIN_RCV
CS_S4_IN_LOGIN: CE_LOGOUT_SESSION_SUCCESS
CS_S4_IN_LOGIN: CE_LOGIN_SUCCESS_SND
CS_S4_IN_LOGIN -> CS_S11_COMPLETE

The state machine processing for idm_state_s4_in_login() doesn't seem to have a
call to untimeout() for CE_LOGOUT_SESSION_SUCCESS

604    case CE_LOGOUT_SESSION_SUCCESS:
605        /*
606         * T8
607         * A session reinstatement request can be received while a
608         * session is active and a login is in process. The iSCSI
609         * connections are shut down by a CE_LOGOUT_SESSION_SUCCESS
610         * event sent from the session to the IDM layer.
611         */
612        if (IDM_CONN_ISTGT(ic)) {
613            ic->ic_transport_ops->it_tgt_conn_disconnect(ic);
614        } else {
615            ic->ic_transport_ops->it_ini_conn_disconnect(ic);
616        }
617        idm_update_state(ic, CS_S11_COMPLETE, event_ctx);
618        break;