Actions
Bug #4772
closed$C prints no output on crash dump following dtrace panic()
Start date:
2014-04-16
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:
Description
Sebastien Roy:
When inducing a panic via dtrace's panic() action, $C produces no output when loading the resulting crash dump in mdb: root@fender.talisker:/var/crash# mdb 0 Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs mpt sd ip hook neti sockfs arp usba stmf stmf_sbd fctl md lofs random idm fcp cpc fcip crypto smbsrv klmmod nfs nsmb ufs logindmux ptm ipc ] > $C >
Adam Leventhal:
I can't reproduce this with dtrace -w -n 'BEGIN{ panic(); }'.
Seb:
I can't reproduce this either within the BEGIN probe, but I can within an fbt probe. For example, dtrace -w -n 'fbt::ip_input:entry{ panic(); }' works nicely for me. I'm also booted from kmdb, but I'm not sure if that's relevant.
Adam:
Walking the stack by hand, things are totally reasonable: > <rsp-8/pnp 0xffffff0007ad7a58: 0xffffff0007ad7ac8 dtrace_panic+0x40 > 0xffffff0007ad7ac8/pnp 0xffffff0007ad7ac8: 0xffffff0007ad7af8 dtrace_action_panic+0x81 > 0xffffff0007ad7af8/pnp 0xffffff0007ad7af8: 0xffffff0007ad7cd8 dtrace_probe+0x764 > 0xffffff0007ad7cd8/pnp 0xffffff0007ad7cd8: 0xffffff0007ad7d18 fbt_invop+0xbf > 0xffffff0007ad7d18/pnp 0xffffff0007ad7d18: 0xffffff0007ad7d68 dtrace_invop+0x54 > 0xffffff0007ad7d68/pnp 0xffffff0007ad7d68: 0xffffff0007ad7ec0 dtrace_invop_callsite > 0xffffff0007ad7ec0/pnp 0xffffff0007ad7ec0: 0xffffff0007ad7f30 cyclic_fire+0x8c > 0xffffff0007ad7f30/pnp 0xffffff0007ad7f30: 0xffffff0007ad7f50 cbe_fire+0x3e > 0xffffff0007ad7f50/pnp 0xffffff0007ad7f50: 0xffffff0007ad7fc0 av_dispatch_autovect+0x95 > 0xffffff0007ad7fc0/pnp 0xffffff0007ad7fc0: 0xffffff0007ad7ff0 dispatch_hilevel+0x20 > 0xffffff0007ad7ff0/pnp 0xffffff0007ad7ff0: 0xffffff0007a05a70 switch_sp_and_call+0x13 > 0xffffff0007a05a70/pnp 0xffffff0007a05a70: 0xffffff0007a05ad0 do_interrupt+0x120 > 0xffffff0007a05ad0/pnp 0xffffff0007a05ad0: 0xffffff0007a05ae0 _interrupt+0xba > 0xffffff0007a05ae0/pnp 0xffffff0007a05ae0: 0xffffff0007a05bd0 mach_cpu_idle+6 > 0xffffff0007a05bd0/pnp 0xffffff0007a05bd0: 0xffffff0007a05c00 cpu_idle+0x11a But take a look at just those frame pointers: 0xffffff0007ad7a58 0xffffff0007ad7ac8 0xffffff0007ad7af8 0xffffff0007ad7cd8 0xffffff0007ad7d18 0xffffff0007ad7d68 0xffffff0007ad7ec0 0xffffff0007ad7f30 0xffffff0007ad7f50 0xffffff0007ad7fc0 0xffffff0007ad7ff0 0xffffff0007a05a70 0xffffff0007a05ad0 0xffffff0007a05ae0 0xffffff0007a05bd0 Notice that when we hit the dtrace_invop_callsite (which is actually a label in invoptrap() before calling dtrace_invop()) the frame pointers all switch from being 16-byte aligned to being merely 8-byte aligned. This violates MDB's check for stack alignment: static int fp_is_aligned(uintptr_t fp, int xpv_panic) { if (!xpv_panic && (fp & (STACK_ALIGN -1))) return (0); if ((fp & sizeof (uintptr_t) - 1)) return (0); return (1); } By nop-ing out the function, $C works: > $C ffffff0007ad7ac8 vpanic() ffffff0007ad7af8 dtrace_action_panic+0x81(ffffff01d9da5588) ffffff0007ad7cd8 dtrace_probe+0x764(21b4, 0, 0, ffffff01cdf15d80, ffffff01cddd56c8, ffffff01cddd56d4) ffffff0007ad7d18 fbt_invop+0xbf(fffffffffb801c30, ffffff0007ad7d78, ffffff01cdd10738) ffffff0007ad7d68 dtrace_invop+0x54(fffffffffb801c30, ffffff0007ad7d78, ffffff01cdd10738) ffffff0007ad7ec0 dtrace_invop_callsite() ffffff0007ad7f30 cyclic_fire+0x8c(fffffffffbc30d20) ffffff0007ad7f50 cbe_fire+0x3e() ffffff0007ad7fc0 av_dispatch_autovect+0x95(d1) ffffff0007ad7ff0 dispatch_hilevel+0x20(d1, 0) ffffff0007a05a70 switch_sp_and_call+0x13() ffffff0007a05ad0 do_interrupt+0x120(ffffff0007a05ae0, 0) ffffff0007a05ae0 _interrupt+0xba() ffffff0007a05bd0 mach_cpu_idle+6() ffffff0007a05c00 cpu_idle+0x11a() ffffff0007a05c20 idle+0xa7() ffffff0007a05c30 thread_start+8() I'd argue both that MDB is being a bit persnickety by insisting that stacks are aligned -- they should be, but the debugger shouldn't completely refuse to proceed. But the invop handler should probably make its call with proper stack-alignment.
Updated by Christopher Siden over 8 years ago
- Status changed from In Progress to Closed
commit 8e50396accfa540cbe0b549256487d237e3d02f3 Author: Adam H. Leventhal <ahl@delphix.com> Date: Mon Apr 21 08:00:44 2014 4772 $C prints no output on crash dump following dtrace panic() Reviewed by: Sebastien Roy <sebastien.roy@delphix.com> Reviewed by: Eric Schrock <eric.schrock@delphix.com> Reviewed by: Robert Mustacchi <rm@joyent.com> Reviewed by: Richard Lowe <richlowe@richlowe.net> Approved by: Eric Schrock <eric.schrock@delphix.com>
Actions