illumos gate

Sebastien Roy:

When inducing a panic via dtrace's panic() action, $C produces no output when
loading the resulting crash dump in mdb:

root@fender.talisker:/var/crash# mdb 0
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp
scsi_vhci zfs mpt sd ip hook neti sockfs arp usba stmf stmf_sbd fctl md lofs
random idm fcp cpc fcip crypto smbsrv klmmod nfs nsmb ufs logindmux ptm ipc ]
> $C
>

Adam Leventhal:

I can't reproduce this with dtrace -w -n 'BEGIN{ panic(); }'.

Seb:

I can't reproduce this either within the BEGIN probe, but I can within an fbt
probe.  For example, dtrace -w -n 'fbt::ip_input:entry{ panic(); }' works
nicely for me.  I'm also booted from kmdb, but I'm not sure if that's relevant.

Adam:

Walking the stack by hand, things are totally reasonable:

> <rsp-8/pnp
0xffffff0007ad7a58:             0xffffff0007ad7ac8
                dtrace_panic+0x40
> 0xffffff0007ad7ac8/pnp
0xffffff0007ad7ac8:             0xffffff0007ad7af8
                dtrace_action_panic+0x81
> 0xffffff0007ad7af8/pnp
0xffffff0007ad7af8:             0xffffff0007ad7cd8
                dtrace_probe+0x764
> 0xffffff0007ad7cd8/pnp
0xffffff0007ad7cd8:             0xffffff0007ad7d18
                fbt_invop+0xbf  
> 0xffffff0007ad7d18/pnp
0xffffff0007ad7d18:             0xffffff0007ad7d68
                dtrace_invop+0x54
> 0xffffff0007ad7d68/pnp
0xffffff0007ad7d68:             0xffffff0007ad7ec0
                dtrace_invop_callsite
> 0xffffff0007ad7ec0/pnp
0xffffff0007ad7ec0:             0xffffff0007ad7f30
                cyclic_fire+0x8c
> 0xffffff0007ad7f30/pnp
0xffffff0007ad7f30:             0xffffff0007ad7f50
                cbe_fire+0x3e   
> 0xffffff0007ad7f50/pnp
0xffffff0007ad7f50:             0xffffff0007ad7fc0
                av_dispatch_autovect+0x95
> 0xffffff0007ad7fc0/pnp
0xffffff0007ad7fc0:             0xffffff0007ad7ff0
                dispatch_hilevel+0x20
> 0xffffff0007ad7ff0/pnp
0xffffff0007ad7ff0:             0xffffff0007a05a70
                switch_sp_and_call+0x13
> 0xffffff0007a05a70/pnp
0xffffff0007a05a70:             0xffffff0007a05ad0
                do_interrupt+0x120
> 0xffffff0007a05ad0/pnp
0xffffff0007a05ad0:             0xffffff0007a05ae0
                _interrupt+0xba 
> 0xffffff0007a05ae0/pnp
0xffffff0007a05ae0:             0xffffff0007a05bd0
                mach_cpu_idle+6 
> 0xffffff0007a05bd0/pnp
0xffffff0007a05bd0:             0xffffff0007a05c00
                cpu_idle+0x11a  

But take a look at just those frame pointers:
0xffffff0007ad7a58
0xffffff0007ad7ac8
0xffffff0007ad7af8
0xffffff0007ad7cd8
0xffffff0007ad7d18
0xffffff0007ad7d68
0xffffff0007ad7ec0
0xffffff0007ad7f30
0xffffff0007ad7f50
0xffffff0007ad7fc0
0xffffff0007ad7ff0
0xffffff0007a05a70
0xffffff0007a05ad0
0xffffff0007a05ae0
0xffffff0007a05bd0

Notice that when we hit the dtrace_invop_callsite (which is actually a label in
invoptrap() before calling dtrace_invop()) the frame pointers all switch from
being 16-byte aligned to being merely 8-byte aligned. This violates MDB's check
for stack alignment:

static int
fp_is_aligned(uintptr_t fp, int xpv_panic)
{
        if (!xpv_panic && (fp & (STACK_ALIGN -1)))
                return (0);
        if ((fp & sizeof (uintptr_t) - 1))
                return (0);
        return (1);
}

By nop-ing out the function, $C works:

> $C
ffffff0007ad7ac8 vpanic()
ffffff0007ad7af8 dtrace_action_panic+0x81(ffffff01d9da5588)
ffffff0007ad7cd8 dtrace_probe+0x764(21b4, 0, 0, ffffff01cdf15d80, 
ffffff01cddd56c8, ffffff01cddd56d4)
ffffff0007ad7d18 fbt_invop+0xbf(fffffffffb801c30, ffffff0007ad7d78, 
ffffff01cdd10738)
ffffff0007ad7d68 dtrace_invop+0x54(fffffffffb801c30, ffffff0007ad7d78, 
ffffff01cdd10738)
ffffff0007ad7ec0 dtrace_invop_callsite()
ffffff0007ad7f30 cyclic_fire+0x8c(fffffffffbc30d20)
ffffff0007ad7f50 cbe_fire+0x3e()
ffffff0007ad7fc0 av_dispatch_autovect+0x95(d1)
ffffff0007ad7ff0 dispatch_hilevel+0x20(d1, 0)
ffffff0007a05a70 switch_sp_and_call+0x13()
ffffff0007a05ad0 do_interrupt+0x120(ffffff0007a05ae0, 0)
ffffff0007a05ae0 _interrupt+0xba()
ffffff0007a05bd0 mach_cpu_idle+6()
ffffff0007a05c00 cpu_idle+0x11a()
ffffff0007a05c20 idle+0xa7()
ffffff0007a05c30 thread_start+8()

I'd argue both that MDB is being a bit persnickety by insisting that stacks are
aligned -- they should be, but the debugger shouldn't completely refuse to
proceed. But the invop handler should probably make its call with proper
stack-alignment.