illumos gate

Many times someone tries to use the ::softstate macro on a variable, but forgets to dereference, leading things to core dump. For example:

[root@magma ~]# mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba xhci stmf_sbd stmf zfs sd mm lofs idm mpt_sas sata i40e cpc logindmux ptm sppp nfs ]
> pcieb_state::walk softstate
Segmentation Fault (core dumped)

If we turn to the core file:

# mdb /zones/global/cores/core.mdb.110812 
mdb: debugger failed with error: fatal signal received
Loading modules: [ libumem.so.1 libc.so.1 libproc.so.1 libavl.so.1 libsysevent.so.1 libnvpair.so.1 libuutil.so.1 ld.so.1 ]
> $C
fffffbffffdfe7d0 genunix.so`soft_state_walk_step+0x20(5c6be0)
fffffbffffdfe810 walk_step+0x7f(5c6be0)
fffffbffffdfe840 walk_common+0x77(5c6be0)
fffffbffffdfe8a0 mdb_pwalk+0xa1(58f2d0, 437680, 0, fffffffffc0e8578)
fffffbffffdfe900 cmd_walk+0xc8(fffffffffc0e8578, 1, 1, 58f5d0)
fffffbffffdfe980 dcmd_invoke+0x7c(5815d0, fffffffffc0e8578, 1, 1, 58f5d0, 0)
fffffbffffdfea20 mdb_call_idcmd+0x112(5815d0, fffffffffc0e8578, 1, 1, 5c6db8, 5c6dd0)
fffffbffffdfeb30 mdb_call+0x449(fffffffffc0e8578, 1, 1)
fffffbffffdfec20 yyparse+0xdf4()
fffffbffffdfed50 mdb_run+0x2cd()
fffffbffffdffae0 main+0xfa1(2, fffffbffffdffb38, fffffbffffdffb50)
fffffbffffdffb10 _start_crt+0x83()
fffffbffffdffb20 _start+0x18()> 5c6be0::print mdb_walk_state_t walk_data | ::print soft_state_walk_t
{
    ssw_ss = {
        array = 0xfffffe25735abd40
        lock = {
            _opaque = [ 0 ]
        }
        size = 0
        n_items = 0
        next = 0xfffffe25735ab6c0
    }
    ssw_pointers = 0
    ssw_index = 0
}

So, we're trying to dereference ssw_pointers which is NULL. This is empty because the number of items and the size is set to zero in the soft state data we read. We should stop the walker from ever getting off the ground by erroring when either size or n_items is zero.

I tested this by trying to reproduce and then do a normal ::walk softstate:

> ::unload genunix
> ::load /var/tmp/genunix.so
> i40e_attach::walk softstate
mdb: softstate size is larger than 1 GiB (0x49d85d8948e06589), invalid softstate?
mdb: failed to perform walk: failed to initialize walk
> pcieb_state::walk softstate
mdb: read invalid softstate: softstate item size is zero
mdb: failed to perform walk: failed to initialize walk
> *pcieb_state::walk softstate
0xfffffe2561b28450
0xfffffe2561b22cd0
0xfffffe25756da808
0xfffffe25756da450
0xfffffe25756da230
0xfffffe25756d0e60
0xfffffe2575858330
0xfffffe257584def0
> *ahci_statep::walk softstate
0xfffffe25a7266d00
0xfffffe25a98aa4c0
> ahci_statep::walk softstate
mdb: read invalid softstate: softstate item size is zero
mdb: failed to perform walk: failed to initialize walk
> asy_soft_state::walk softstate
mdb: read invalid softstate: softstate item size is zero
mdb: failed to perform walk: failed to initialize walk
> *asy_soft_state::walk softstate
0xfffffe256465f4f0
0xfffffe256465f3f8