Project

General

Profile

Actions

Bug #3919

closed

mdb/pmap should show unresolved map names on core dumps

Added by Robert Mustacchi almost 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Normal
Category:
-
Start date:
2013-07-28
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

% LONGPATH=`tail -40 /usr/dict/words | grep -v \\' | nawk '{ printf("%s/", $0)}'`
% mkdir -p $LONGPATH
% cp /usr/lib/libumem.so.1 $LONGPATH
% export LD_LIBRARY_PATH=$LONGPATH
% export LD_PRELOAD=libumem.so.1
% cat &
% gcore `pgrep -n cat`
% pkill -9 -n cat

Take the core file that results from the gcore and note that pmap and mdb both behave correctly:

% pmap core.40191
...
FEF10000     196K r-x-- /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/zigging/zigzag/zigzagging/zilch/Zimmerman/zinc/zing/Zion/Zionism/zip/zippy/zircon/zirconium/zloty/zodiac/zodiacal/Zoe/Zomba/zombie/zone/zoo/zoology/zoom/Zorn/Zoroaster/Zoroastrian/zounds/zucchini/Zurich/zygote/libumem.so.1
FEF51000      76K rw--- /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/zigging/zigzag/zigzagging/zilch/Zimmerman/zinc/zing/Zion/Zionism/zip/zippy/zircon/zirconium/zloty/zodiac/zodiacal/Zoe/Zomba/zombie/zone/zoo/zoology/zoom/Zorn/Zoroaster/Zoroastrian/zounds/zucchini/Zurich/zygote/libumem.so.1
FEF64000      24K rw--- /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/zigging/zigzag/zigzagging/zilch/Zimmerman/zinc/zing/Zion/Zionism/zip/zippy/zircon/zirconium/zloty/zodiac/zodiacal/Zoe/Zomba/zombie/zone/zoo/zoology/zoom/Zorn/Zoroaster/Zoroastrian/zounds/zucchini/Zurich/zygote/libumem.so.1
...

% mdb core.40191
> $m
...
fef10000 fef41000    31000 /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/zigging/zigzag/zigzagging/zilch/Zimmerman/zinc/zing/Zion/Zionism/zip/zippy/zircon/zirconium/zloty/zodiac/zodiacal/Zoe/Zomba/zombie/zone/zoo/zoology/zoom/Zorn/Zoroaster/Zoroastrian/zounds/zucchini/Zurich/zygote/libumem.so.1
fef51000 fef64000    13000 /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/zigging/zigzag/zigzagging/zilch/Zimmerman/zinc/zing/Zion/Zionism/zip/zippy/zircon/zirconium/zloty/zodiac/zodiacal/Zoe/Zomba/zombie/zone/zoo/zoology/zoom/Zorn/Zoroaster/Zoroastrian/zounds/zucchini/Zurich/zygote/libumem.so.1
fef64000 fef6a000     6000 /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/zigging/zigzag/zigzagging/zilch/Zimmerman/zinc/zing/Zion/Zionism/zip/zippy/zircon/zirconium/zloty/zodiac/zodiacal/Zoe/Zomba/zombie/zone/zoo/zoology/zoom/Zorn/Zoroaster/Zoroastrian/zounds/zucchini/Zurich/zygote/libumem.so.1
...

Now delete the libumem.so.1:

% rm $LONGPATH/libumem.so.1

Now pmap won't find any libumem.so.1 at all:

# pmap core.40191
core 'core.40191' of 40191:     cat
08046000       8K rw---    [ stack ]
08050000      68K r-x--  /opt/local/bin/gcat
08070000       8K rwx--  /opt/local/bin/gcat
08072000     320K rwx--    [ heap ]
FEA30000      64K rwx--    [ anon ]
FEB3E000       4K rw---    [ anon ]
FEB40000       4K r-x--  /lib/libumem_trampoline.so.1
FEB51000       4K rw---  /lib/libumem_trampoline.so.1
FEB60000      12K rwx--  /lib/libumem_trampoline.so.1
FEBD0000      64K rw---    [ anon ]
FEBE7000     128K rw---    [ anon ]
FEC08000       4K rwx--    [ anon ]
FEC0C000      76K r----    [ anon ]
FEC20000       4K rwx--    [ anon ]
FEC30000      24K rwx--    [ anon ]
FEC40000     100K r-x--  /opt/local/gcc47/i386-sun-solaris2.11/lib/libgcc_s.so.1
FEC68000       4K rwx--  /opt/local/gcc47/i386-sun-solaris2.11/lib/libgcc_s.so.1
FEC70000       4K rwx--    [ anon ]
FEC80000    1232K r-x--  /lib/libc.so.1
FEDC4000      36K rwx--  /lib/libc.so.1
FEDCD000       8K rwx--  /lib/libc.so.1
FEDD0000     904K r-x--  /opt/local/lib/libiconv.so.2.5.1
FEEC1000       8K rwx--  /opt/local/lib/libiconv.so.2.5.1
FEED0000       4K rwx--    [ anon ]
FEEE0000      36K r-x--  /opt/local/lib/libintl.so.8.1.1
FEEF8000       4K rwx--  /opt/local/lib/libintl.so.8.1.1
FEF00000       4K rwx--    [ anon ]
FEF10000     196K r-x--    [ anon ]
FEF51000      76K rw---    [ anon ]
FEF64000      24K rw---    [ anon ]
FEF70000       4K r----*   [ anon ]
FEF80000       4K rw---    [ anon ]
FEF90000       4K rw---    [ anon ]
FEFA0000       4K rwx--    [ anon ]
FEFB0000       4K rwx--    [ anon ]
FEFB6000     212K r-x--  /lib/ld.so.1
FEFFB000       8K rwx--  /lib/ld.so.1
FEFFD000       4K rwx--  /lib/ld.so.1
 total      3676K

mdb has the path, but it's been truncated to PRMAPSZ (64 bytes):

> $m
    BASE    LIMIT     SIZE NAME
        2000 [ stack ]
       11000 /opt/local/bin/gcat
        2000 /opt/local/bin/gcat
       50000 [ heap ]
fea30000 fea40000    10000 
feb3e000 feb3f000     1000 
feb40000 feb41000     1000 LMfef00428`/lib/libumem_trampoline.so.1
feb51000 feb52000     1000 LMfef00428`/lib/libumem_trampoline.so.1
feb60000 feb63000     3000 LMfef00428`/lib/libumem_trampoline.so.1
febd0000 febe0000    10000 
febe7000 fec07000    20000 
fec08000 fec09000     1000 
fec0c000 fec1f000    13000 
fec20000 fec21000     1000 
fec30000 fec36000     6000 
fec40000 fec59000    19000 
/opt/local/gcc47/i386-sun-solaris2.11/lib/libgcc_s.so.1
fec68000 fec69000     1000 
/opt/local/gcc47/i386-sun-solaris2.11/lib/libgcc_s.so.1
fec70000 fec71000     1000 
fec80000 fedb4000   134000 /lib/libc.so.1
fedc4000 fedcd000     9000 /lib/libc.so.1
fedcd000 fedcf000     2000 /lib/libc.so.1
fedd0000 feeb2000    e2000 /opt/local/lib/libiconv.so.2.5.1
feec1000 feec3000     2000 /opt/local/lib/libiconv.so.2.5.1
feed0000 feed1000     1000 
feee0000 feee9000     9000 /opt/local/lib/libintl.so.8.1.1
feef8000 feef9000     1000 /opt/local/lib/libintl.so.8.1.1
fef00000 fef01000     1000 
fef10000 fef41000    31000 /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/ziggi
fef51000 fef64000    13000 /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/ziggi
fef64000 fef6a000     6000 /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/ziggi
fef70000 fef71000     1000 
fef80000 fef81000     1000 
fef90000 fef91000     1000 
fefa0000 fefa1000     1000 
fefb0000 fefb1000     1000 
fefb6000 fefeb000    35000 /lib/ld.so.1
feffb000 feffd000     2000 /lib/ld.so.1
feffd000 feffe000     1000 /lib/ld.so.1

But mdb won't find any libumem.so.1 symbols:

> ::nm ! grep umem_alloc
mdb: unable to dump symbol data for: /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/ziggi

There are multiple ways to fix this issue, but by far the simplest (and most broadly useful) is to fix this issue by fixing a broader issue: that mdb (and pmap) try to resolve linker names on a core dump. Both of these commands have options for the unresolved behavior ("-l" for pmap and the extraordinarily obscure "lmraw" option for mdb); these options should be enabled by default when debugging postmortem. When this is done, the $m for the above dump has the true name length:

BASE    LIMIT     SIZE NAME
 8046000  8048000     2000 [ stack ]
 8050000  8061000    11000 /opt/local/bin/gcat
 8070000  8072000     2000 /opt/local/bin/gcat
 8072000  80c2000    50000 [ heap ]
fea30000 fea40000    10000 
feb3e000 feb3f000     1000 
feb40000 feb41000     1000 LMfef00428`/lib/libumem_trampoline.so.1
feb51000 feb52000     1000 LMfef00428`/lib/libumem_trampoline.so.1
feb60000 feb63000     3000 LMfef00428`/lib/libumem_trampoline.so.1
febd0000 febe0000    10000 
febe7000 fec07000    20000 
fec08000 fec09000     1000 
fec0c000 fec1f000    13000 
fec20000 fec21000     1000 
fec30000 fec36000     6000 
fec40000 fec59000    19000 
/opt/local/gcc47/i386-sun-solaris2.11/lib/libgcc_s.so.1
fec68000 fec69000     1000 
/opt/local/gcc47/i386-sun-solaris2.11/lib/libgcc_s.so.1
fec70000 fec71000     1000 
fec80000 fedb4000   134000 /lib/libc.so.1
fedc4000 fedcd000     9000 /lib/libc.so.1
fedcd000 fedcf000     2000 /lib/libc.so.1
fedd0000 feeb2000    e2000 /opt/local/lib/libiconv.so.2.5.1
feec1000 feec3000     2000 /opt/local/lib/libiconv.so.2.5.1
feed0000 feed1000     1000 
feee0000 feee9000     9000 /opt/local/lib/libintl.so.8.1.1
feef8000 feef9000     1000 /opt/local/lib/libintl.so.8.1.1
fef00000 fef01000     1000 
fef10000 fef41000    31000 /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/zigging/zigzag/zigzagging/zilch/Zimmerman/zinc/zing/Zion/Zionism/zip/zippy/zircon/zirconium/zloty/zodiac/zodiacal/Zoe/Zomba/zombie/zone/zoo/zoology/zoom/Zorn/Zoroaster/Zoroastrian/zounds/zucchini/Zurich/zygote/libumem.so.1
fef51000 fef64000    13000 /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/zigging/zigzag/zigzagging/zilch/Zimmerman/zinc/zing/Zion/Zionism/zip/zippy/zircon/zirconium/zloty/zodiac/zodiacal/Zoe/Zomba/zombie/zone/zoo/zoology/zoom/Zorn/Zoroaster/Zoroastrian/zo
unds/zucchini/Zurich/zygote/libumem.so.1
fef64000 fef6a000     6000 /root/zero/zeroes/zeroth/zest/zesty/zeta/Zeus/Ziegler/zig/zigging/zigzag/zigzagging/zilch/Zimmerman/zinc/zing/Zion/Zionism/zip/zippy/zircon/zirconium/zloty/zodiac/zodiacal/Zoe/Zomba/zombie/zone/zoo/zoology/zoom/Zorn/Zoroaster/Zoroastrian/zo
unds/zucchini/Zurich/zygote/libumem.so.1
fef70000 fef71000     1000 
fef80000 fef81000     1000 
fef90000 fef91000     1000 
fefa0000 fefa1000     1000 
fefb0000 fefb1000     1000 
fefb6000 fefeb000    35000 /lib/ld.so.1
feffb000 feffd000     2000 /lib/ld.so.1
feffd000 feffe000     1000 /lib/ld.so.1
Actions #1

Updated by Robert Mustacchi almost 8 years ago

  • Status changed from New to Resolved
  • % Done changed from 90 to 100

Resolved in 8f68126c6dc4f2178929233f0c7ccce877e973ca.

Actions

Also available in: Atom PDF