Project

General

Profile

Actions

Bug #14845

open

ena panic during first AMI boot

Added by Andy Fiddaman 2 months ago. Updated 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
driver - device drivers
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:

Description

When deploying an OmniOS AMI to AWS EC2 (I am testing with a t3.micro instance), it always panics on the first couple of boots and then seems to work ok after that. Since there's a serial console, it's much easier to grab things than it used to be:

panic[cpu1]/thread=fffffe0002b3dc20: timed out waiting for admin response

Warning - stack not written to the dump buffer
fffffe0002b3d350 ena:ena_debug+37cca5ec ()
fffffe0002b3d3d0 ena:ena_admin_get_basic_stats+7a ()
fffffe0002b3d460 ena:ena_stat_device_basic_update+39 ()
fffffe0002b3d4c0 ena:ena_m_stat+3f ()
fffffe0002b3d500 mac:mac_stat_get+4a ()
fffffe0002b3d530 mac:mac_client_ifspeed+24 ()
fffffe0002b3d550 mac:mac_client_stat_get+71 ()
fffffe0002b3d5b0 mac:mac_compute_soft_ring_count+111 ()
fffffe0002b3d610 mac:mac_flow_cpu_init+46 ()
fffffe0002b3d6c0 mac:mac_fanout_setup+6c ()
fffffe0002b3d740 mac:mac_srs_group_setup+c8 ()
fffffe0002b3d820 mac:mac_datapath_setup+6bf ()
fffffe0002b3d8d0 mac:mac_client_datapath_setup+290 ()
fffffe0002b3d970 mac:i_mac_unicast_add+4e1 ()
fffffe0002b3d9f0 mac:mac_unicast_add+73 ()
fffffe0002b3da30 dls:dls_mac_active_set+69 ()
fffffe0002b3da60 dls:dls_active_set+54 ()
fffffe0002b3db00 dld:proto_bind_req+91 ()
fffffe0002b3db20 dld:dld_proto+105 ()
fffffe0002b3db60 dld:dld_wput_nondata_task+79 ()
fffffe0002b3dc00 genunix:taskq_d_thread+1ac ()
fffffe0002b3dc10 unix:thread_start+b ()

panic: entering debugger (no dump device, continue to reboot)

Welcome to kmdb
kmdb: unable to determine terminal type: assuming `vt100'
Loaded modules: [ scsi_vhci stmf_sbd crypto mac cpc uppc mr_sas neti sd ptm ufs unix mpt zfs krtld s1394 sata apix uhci hook pmcs lofs genunix ip logindmux nsmb usba xhci specfs pcplusmp nfs random mm cpu.generic arp mpt_sas stmf emlxs sockfs smbsrv ]
[0]> $Cfffffffffbc8e3a0 kmdb_enter+0xb()
fffffffffbc8e3d0 debug_enter+0x75(fffffffffb9413b0)
fffffffffbc8e4c0 panicsys+0x616(fffffffff8324c58, fffffe0001f342b8, fffffffffbc8e4d0, 1)
fffffe0001f342a0 vpanic+0x15c()
fffffe0001f34310 0xfffffffffb8a9ad1()
fffffe0001f34350 0xfffffffff831f0cc()
fffffe0001f343d0 ena`ena_admin_get_basic_stats+0x7a(fffffe01e359f6c0, fffffe0001f343f8)
fffffe0001f34460 ena`ena_stat_device_basic_update+0x39(fffffe01e19998e0, 0)
fffffe0001f344c0 ena`ena_m_stat+0x3f(fffffe01e359f6c0, 3e8, fffffe0001f344d0)
fffffe0001f34500 mac`mac_stat_get+0x4a(fffffe01e35f8008, 3e8)
...

Here's the bit of the system log from ena_debug too

NOTICE: ena0: running attach sequence: PCI config (0)
NOTICE: ena0: vid: 0x1d0f did: 0xec20 rev: 0x0 svid: 0x0 sdid: 0x0
NOTICE: ena0: attach sequence completed: PCI config (0)
NOTICE: ena0: running attach sequence: BAR mapping (1)
NOTICE: ena0: register size: 16384
NOTICE: ena0: registers mapped to base: 0xfffffe0198d11000
NOTICE: ena0: attach sequence completed: BAR mapping (1)
NOTICE: ena0: running attach sequence: device initialization (2)
NOTICE: ena0: impl ID: 1
NOTICE: ena0: device version: 10
NOTICE: ena0: supported features: 0x14005886
NOTICE: ena0: phys addr width: 48
NOTICE: ena0: virt addr width: 64
NOTICE: ena0: mac addr: a:f7:37:54:f3:0
NOTICE: ena0: max MTU: 9216
NOTICE: ena0: LINK CHANGE supported: Y enabled: Y
NOTICE: ena0: FATAL ERROR supported: Y enabled: Y
NOTICE: ena0: WARNING supported: Y enabled: Y
NOTICE: ena0: NOTIFICATION supported: Y enabled: Y
NOTICE: ena0: KEEP ALIVE supported: Y enabled: N
NOTICE: ena0: REFRESH CAPABILITIES supported: Y enabled: N
NOTICE: ena0: attach sequence completed: device initialization (2)
NOTICE: ena0: running attach sequence: ena.conf (3)
NOTICE: ena0: attach sequence completed: ena.conf (3)
NOTICE: ena0: running attach sequence: device config (4)
NOTICE: ena0: attach sequence completed: device config (4)
NOTICE: ena0: running attach sequence: interrupt allocation (5)
NOTICE: ena0: supported interrupt types: 0x5
NOTICE: ena0: 9 MSI-X interrupts available
NOTICE: ena0: MSI-X interrupts allocated: 2, cap: 0x32, pri: 6
NOTICE: ena0: attach sequence completed: interrupt allocation (5)
NOTICE: ena0: running attach sequence: interrupt handlers (6)
NOTICE: ena0: attach sequence completed: interrupt handlers (6)
NOTICE: ena0: running attach sequence: Tx queues (7)
NOTICE: ena0: attach sequence completed: Tx queues (7)
NOTICE: ena0: running attach sequence: Rx queues (8)
NOTICE: ena0: attach sequence completed: Rx queues (8)
NOTICE: ena0: running attach sequence: mac registration (9)
NOTICE: ena0 registered
NOTICE: ena0: attach sequence completed: mac registration (9)
NOTICE: ena0: running attach sequence: enable interrupts (10)
NOTICE: ena0: attach sequence completed: enable interrupts (10)
NOTICE: ena0: AENQ Group: (0x0) LINK CHANGE Syndrome: 0x0 ts: 98775604607 us
[1]> ena_admin_cmd_timeout_ns/J
ena`ena_admin_cmd_timeout_ns:
ena`ena_admin_cmd_timeout_ns:   1dcd6500 
[1]> fffffe01deef1000::print ena_t
{
    ena_dip = 0xfffffe01de61f5a8
    ena_instance = 0
    ena_lock = {
        _opaque = [ 0 ]
    }
    ena_attach_seq = 0t11 (ENA_ATTACH_INTRS_ENABLE)
    ena_state = 0x2
    ena_pci_hdl = 0xfffffe01df117140
    ena_reg_size = 0x4000
    ena_reg_base = 0xfffffe0198d11000
    ena_reg_attr = {
        devacc_attr_version = 0x2
        devacc_attr_endian_flags = 0
        devacc_attr_dataorder = 0
        devacc_attr_access = 0
    }
    ena_reg_hdl = 0xfffffe01df117000
    ena_pci_vid = 0x1d0f
    ena_pci_did = 0xec20
    ena_pci_rev = 0
    ena_pci_svid = 0
    ena_pci_sdid = 0
    ena_dev_major_vsn = 0             
    ena_dev_minor_vsn = 0xa
    ena_ctrl_major_vsn = 0
    ena_ctrl_minor_vsn = 0
    ena_ctrl_subminor_vsn = 0x1
    ena_ctrl_impl_id = 0x1
    ena_num_intrs = 0x2
    ena_intr_handles = 0xfffffe01dfe63ba0
    ena_intr_handles_sz = 0x10
    ena_intr_caps = 0x32
    ena_intr_pri = 0x6
    ena_mh = 0xfffffe01e1d86008
    ena_page_sz = 0x1000
    ena_mtu = 0x2400
    ena_max_frame_hdr = 0x12
    ena_max_frame_total = 0x2412
    ena_tx_buf_sz = 0x3000
    ena_rx_buf_sz = 0x3000
    ena_tx_sgl_max_sz = 0x1
    ena_rx_sgl_max_sz = 0x1
    ena_rxq_num_descs = 0x400
    ena_txq_num_descs = 0x400
    ena_rxq_intr_limit = 0x100
    ena_rxqs = 0xfffffe01dfbf8a80     
    ena_num_rxqs = 0x1
    ena_txqs = 0xfffffe01dfbf8d00
    ena_num_txqs = 0x1
    ena_device_basic_kstat = 0xfffffe01dfc4daa0
    ena_device_extended_kstat = 0xfffffe01dfc4dcd0
    ena_aenq_stat = {
        eaes_default = {
            name = [ "default" ]
            data_type = 0x4
            value = {
                c = [ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0' ]
                i32 = 0
                ui32 = 0
                str = {
                    addr = {
                        ptr = 0
                        ptr32 = 0
                        __pad = [ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0' ]
                    }
                    len = 0
                }
                i64 = 0
                ui64 = 0              
                l = 0
                ul = 0
                ll = 0
                ull = 0
                f = <FLOAT>
                d = <FLOAT>
            }
        }
        eaes_link_change = {
            name = [ "link_change" ]
            data_type = 0x4
            value = {
                c = [ '\001', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0' ]
                i32 = 0x1
                ui32 = 0x1
                str = {
                    addr = {
                        ptr = 1
                        ptr32 = 0x1
                        __pad = [ '\001', '\0', '\0', '\0', '\0', '\0', '\0', '\0' ]
                    }
                    len = 0
                }                     
                i64 = 0x1
                ui64 = 0x1
                l = 0x1
                ul = 0x1
                ll = 0x1
                ull = 0x1
                f = <FLOAT>
                d = <FLOAT>
            }
        }
    }
    ena_aenq_kstat = 0xfffffe01dfc4ded0
    ena_aq = {
        ea_sq_lock = {
            _opaque = [ 0 ]
        }
        ea_cq_lock = {
            _opaque = [ 0 ]
        }
        ea_stat_lock = {
            _opaque = [ 0 ]
        }
        ea_cmd_timeout_ns = 0x1dcd6500
        ea_qlen = 0x20
        ea_poll_mode = 0x1 (B_TRUE)
        ea_cmd_ctxs = 0xfffffe01dfbf9ac0
        ea_cmd_ctxs_free = {
            list_size = 0x28
            list_offset = 0
            list_head = {
                list_next = 0xfffffe01dfbf9ae8
                list_prev = 0xfffffe01dfbf9f98
            }
        }
        ea_pending_cmds = 0x1
        ea_sq = {
            eas_entries = 0xfffffe01dfe65020
            eas_dma = {
                edb_va = 0xfffffe01dfe65020
                edb_len = 0x800
                edb_real_len = 0x800
                edb_used_len = 0
                edb_acc_hdl = 0xfffffe01df63b180
                edb_dma_hdl = 0xfffffe01df131ac0
                edb_cookie = 0xfffffe01df131cd0
            }                         
            eas_dbaddr = 0xfffffe0198d1102c
            eas_tail = 0x4e
            eas_phase = 0x1
        }
        ea_cq = {
            eac_entries = 0xfffffe01dfe66020
            eac_dma = {
                edb_va = 0xfffffe01dfe66020
                edb_len = 0x800
                edb_real_len = 0x800
                edb_used_len = 0
                edb_acc_hdl = 0xfffffe01df63b040
                edb_dma_hdl = 0xfffffe01df131080
                edb_cookie = 0xfffffe01df131290
            }
            eac_head = 0x4d
            eac_phase = 0x1
        }
        ea_stats = {
            cmds_fail = 0
            cmds_submitted = 0x4e
            cmds_success = 0x4d
            queue_full = 0            
        }
    }
    ena_aenq = {
        eaenq_descs = 0xfffffe01dfe67020
        eaenq_dma = {
            edb_va = 0xfffffe01dfe67020
            edb_len = 0x800
            edb_real_len = 0x800
            edb_used_len = 0
            edb_acc_hdl = 0xfffffe01dfe5de40
            edb_dma_hdl = 0xfffffe01dfc85540
            edb_cookie = 0xfffffe01dfc85750
        }
        eaenq_hdlrs = [ ena`ena_aenq_link_change_hdlr, ena`ena_aenq_default_hdlr, ena`ena_aenq_default_hdlr, ena`ena_aenq_default_hdlr, ena`ena_aenq_default_hdlr, ena`ena_aenq_default_hdlr ]
        eaenq_num_descs = 0x20
        eaenq_head = 0x1
        eaenq_phase = 0x1
    }
    ena_host_info = {
        edb_va = 0xfffffe01dfe68000
        edb_len = 0x1000
        edb_real_len = 0x1000
        edb_used_len = 0              
        edb_acc_hdl = 0xfffffe01dfe5dd00
        edb_dma_hdl = 0xfffffe01dfc84b00
        edb_cookie = 0xfffffe01dfc84d10
    }
    ena_supported_features = 0x14005886
    ena_dma_width = 0x30
    ena_link_up = 0x1 (B_TRUE)
    ena_link_autoneg = 0x1 (B_TRUE)
    ena_link_full_duplex = 0 (0)
    ena_link_duplex = 0x2 (LINK_DUPLEX_FULL)
    ena_link_speed_mbits = 0xf4240
    ena_link_speeds = 0x1 (ENAHW_LINK_SPEED_1G)
    ena_link_state = 0 (LINK_STATE_DOWN)
    ena_aenq_supported_groups = 0x3f
    ena_aenq_enabled_groups = 0xf
    ena_tx_max_sq_num = 0x8
    ena_tx_max_sq_num_descs = 0x400
    ena_tx_max_cq_num = 0x8
    ena_tx_max_cq_num_descs = 0x400
    ena_tx_max_desc_per_pkt = 0x11
    ena_tx_max_hdr_len = 0x80
    ena_rx_max_sq_num = 0x8
    ena_rx_max_sq_num_descs = 0x400   
    ena_rx_max_cq_num = 0x8
    ena_rx_max_cq_num_descs = 0x400
    ena_rx_max_desc_per_pkt = 0x1c
    ena_max_io_queues = 0x1
    ena_tx_l3_ipv4_csum = 0x1 (B_TRUE)
    ena_tx_l4_ipv4_part_csum = 0x1 (B_TRUE)
    ena_tx_l4_ipv4_full_csum = 0 (0)
    ena_tx_l4_ipv4_lso = 0 (0)
    ena_tx_l4_ipv6_part_csum = 0 (0)
    ena_tx_l4_ipv6_full_csum = 0 (0)
    ena_tx_l4_ipv6_lso = 0 (0)
    ena_rx_l3_ipv4_csum = 0 (0)
    ena_rx_l4_ipv4_csum = 0 (0)
    ena_rx_l4_ipv6_csum = 0 (0)
    ena_rx_hash = 0 (0)
    ena_max_mtu = 0x2400
    ena_mac_addr = [ 0xa, 0xf7, 0x37, 0x54, 0xf3, 0 ]
}

Related issues

Related to illumos gate - Feature #13689: Want AWS ENA driverClosedRyan Zezeski

Actions
Actions #1

Updated by Andy Fiddaman 2 months ago

Actions #2

Updated by Andy Fiddaman 2 months ago

  • Description updated (diff)
Actions #3

Updated by Andy Fiddaman 2 months ago

  • Description updated (diff)
Actions

Also available in: Atom PDF