Project

General

Profile

Actions

Bug #3866

closed

panic in idm module

Added by Christopher Siden almost 9 years ago. Updated almost 9 years ago.

Status:
Closed
Priority:
Normal
Category:
comstar - iSCSI/FC/SAS target
Start date:
2013-07-03
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

Analysis by Jeff Biseda of a panic we hit at Delphix:

> $C
ffffff007b4c69e0 vpanic()
ffffff007b4c6a10 mutex_panic+0x73(fffffffffb961bd0, ffffff120f9fa2e0)
ffffff007b4c6a70 mutex_vector_enter+0x446(ffffff120f9fa2e0)
ffffff007b4c6ac0 idm_conn_event+0x36(ffffff120f9f9000, 12, 0)
ffffff007b4c6ae0 idm_login_timeout+0x1d(ffffff120f9f9000)
ffffff007b4c6b20 callout_list_expire+0x77(ffffff113dda8fc0, ffffff11b6480100)
ffffff007b4c6b50 callout_expire+0x31(ffffff113dda8fc0)
ffffff007b4c6b70 callout_execute+0x1e(ffffff113dda8fc0)
ffffff007b4c6c20 taskq_thread+0x285(ffffff1159675490)
ffffff007b4c6c30 thread_start+8()

Is the idm_conn_t still around?

> ffffff120f9fa2e0::whatis
ffffff120f9fa2e0 is ffffff120f9f9000+12e0, freed from kmem_alloc_8192

No, it's been freed, and indeed, it's not in the global list.

> ::iscsi_conn
idm_conn_t       Type   Transport     State/FFP
ffffff120ba8f000 Tgt    SOCKETS         5/1     
ffffff135e250000 Tgt    SOCKETS         5/1     
ffffff115a05b000 Tgt    SOCKETS         5/1     
ffffff135fdc8000 Tgt    SOCKETS         5/1     
ffffff1359eba000 Tgt    SOCKETS         5/1  

We're being called back here from a routine queued by timeout().

I see numerous calls to untimeout() at various places in the state machine.
There must be some transition which is failing to call untimeout().
I parsed through the audit buffers and this is the history of the state machine
transitions that we went through for this connection:

CS_S1_FREE: CE_CONNECT_ACCEPT
SC_S1_FREE -> SC_S3_XPT_UP
SC_S3_XPT_UP: CE_LOGIN_RCV
SC_S3_XPT_UP -> CS_S4_IN_LOGIN
CS_S4_IN_LOGIN: CE_MISC_TX
CS_S4_IN_LOGIN: CE_LOGIN_RCV
CS_S4_IN_LOGIN: CE_MISC_TX
CS_S4_IN_LOGIN: CE_LOGIN_RCV
CS_S4_IN_LOGIN: CE_LOGOUT_SESSION_SUCCESS
CS_S4_IN_LOGIN: CE_LOGIN_SUCCESS_SND
CS_S4_IN_LOGIN -> CS_S11_COMPLETE
The state machine processing for idm_state_s4_in_login() doesn't seem to have a
call to untimeout() for CE_LOGOUT_SESSION_SUCCESS

604    case CE_LOGOUT_SESSION_SUCCESS:
605        /*
606         * T8
607         * A session reinstatement request can be received while a
608         * session is active and a login is in process. The iSCSI
609         * connections are shut down by a CE_LOGOUT_SESSION_SUCCESS
610         * event sent from the session to the IDM layer.
611         */
612        if (IDM_CONN_ISTGT(ic)) {
613            ic->ic_transport_ops->it_tgt_conn_disconnect(ic);
614        } else {
615            ic->ic_transport_ops->it_ini_conn_disconnect(ic);
616        }
617        idm_update_state(ic, CS_S11_COMPLETE, event_ctx);
618        break;
Actions #1

Updated by Christopher Siden almost 9 years ago

  • Status changed from In Progress to Closed
commit 57ff5e7
Author: Jeff Biseda <jeff.biseda@delphix.com>
Date:   Mon Jul 8 09:57:02 2013

    3866 panic in idm module
    3867 stmfCreateLu failed: GUID_IN_USE
    3868 iscsi target not accepting any new connections
    Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
    Reviewed by: Jeremy Jones <jeremy@delphix.com>
    Reviewed by: Eric Diven <eric.diven@delphix.com>
    Reviewed by: Richard Lowe <richlowe@richlowe.net>
    Reviewed by: T Nguyen <truongqnguien@gmail.com>
    Approved by: Gordon Ross <gwr@nexenta.com>
Actions

Also available in: Atom PDF