Actions
Bug #3987
closed
svc.startd dies in utmpx_postfork()
Start date:
2013-08-04
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:
Description
A thoth analyzer was written to look at threads in the process:
#
# This analyzer only applies to core files
#
if [ "$THOTH_TYPE" != "core" ]; then
exit 0;
fi
#
# This is only OS-2359 if we have utmpx_postfork in our stack
#
if ( ! echo ::stack | mdb $THOTH_DUMP | grep utmpx_postfork > /dev/null ); then
exit 0;
fi
#
# We're interested in this dump; emit all stacks
#
echo ::stacks | mdb $THOTH_DUMP
Here's the output of that:
$ thoth analyzer OS-2359-stacks < OS-2359-stacks
thoth: reading analyzer 'OS-2359-stacks' from stdin
thoth: added analyzer 'OS-2359-stacks'
$ thoth analyze cmd=svc.startd OS-2359-stacks
=== 40214dd1db0d0cbe03b8c1b07ed5caf1 ===
THREAD STATE SOBJ COUNT
1 UNPARKED <NONE> 1
libc.so.1`mutex_lock_impl+0x291
libc.so.1`mutex_lock+0x19
libumem.so.1`vmem_alloc+0x10f
libumem.so.1`umem_slab_create+0x78
libumem.so.1`umem_slab_alloc+0x8a
libumem.so.1`umem_cache_alloc+0x140
libumem.so.1`umem_alloc+0x50
libumem.so.1`malloc+0x36
libuutil.so.1`uu_zalloc+0x1e
libuutil.so.1`uu_list_pool_create+0x7e
wait_init+0x6e
startup+0x201
main+0x1e4
_start+0x83
2 UNPARKED <NONE> 1
libc.so.1`raise+0x2b
libc.so.1`abort+0x10e
utmpx_postfork+0x44
fork_common+0x186
fork_configd+0x8d
fork_configd_thread+0x2ca
libc.so.1`_thrp_setup+0x88
libc.so.1`_lwp_start
=== ef4b91b304ee21607dae77a7a5f770b8 ===
THREAD STATE SOBJ COUNT
2 UNPARKED <NONE> 1
libc.so.1`raise+0x2b
libc.so.1`abort+0x10e
utmpx_postfork+0x44
fork_common+0x186
fork_configd+0x8d
fork_configd_thread+0x2ca
libc.so.1`_thrp_setup+0x88
libc.so.1`_lwp_start
=== e7d8f040512051792368d6bf1a4389b4 ===
THREAD STATE SOBJ COUNT
1 UNPARKED <NONE> 1
libc.so.1`mutex_lock_impl+0x291
libc.so.1`mutex_lock+0x19
libumem.so.1`umem_cache_alloc+0x5a
libumem.so.1`umem_alloc+0x50
libumem.so.1`malloc+0x36
libuutil.so.1`uu_zalloc+0x1e
libuutil.so.1`uu_list_pool_create+0x7e
wait_init+0x6e
startup+0x201
main+0x1e4
_start+0x83
2 UNPARKED <NONE> 1
libc.so.1`raise+0x2b
libc.so.1`abort+0x10e
utmpx_postfork+0x44
fork_common+0x186
fork_configd+0x8d
fork_configd_thread+0x2ca
libc.so.1`_thrp_setup+0x88
libc.so.1`_lwp_start
=== 000c3a37469868df1eb1c74f60ebbf25 ===
THREAD STATE SOBJ COUNT
1 UNPARKED <NONE> 1
libc.so.1`mutex_lock_impl+0x291
libc.so.1`mutex_lock+0x19
libumem.so.1`vmem_alloc+0x10f
libumem.so.1`umem_slab_create+0x78
libumem.so.1`umem_slab_alloc+0x8a
libumem.so.1`umem_cache_alloc+0x140
libumem.so.1`umem_alloc+0x50
libumem.so.1`malloc+0x36
libuutil.so.1`uu_zalloc+0x1e
libuutil.so.1`uu_list_pool_create+0x7e
wait_init+0x6e
startup+0x201
main+0x1e4
_start+0x83
2 UNPARKED <NONE> 1
libc.so.1`raise+0x2b
libc.so.1`abort+0x10e
utmpx_postfork+0x44
fork_common+0x186
fork_configd+0x8d
fork_configd_thread+0x2ca
libc.so.1`_thrp_setup+0x88
libc.so.1`_lwp_start
=== e40609350344472fce251e9d6c10440c ===
THREAD STATE SOBJ COUNT
1 UNPARKED <NONE> 1
libc.so.1`mutex_lock_impl+0x291
libc.so.1`mutex_lock+0x19
libumem.so.1`vmem_alloc+0x10f
libumem.so.1`umem_slab_create+0x78
libumem.so.1`umem_slab_alloc+0x8a
libumem.so.1`umem_cache_alloc+0x140
libumem.so.1`umem_alloc+0x50
libumem.so.1`malloc+0x36
libuutil.so.1`uu_zalloc+0x1e
libuutil.so.1`uu_list_pool_create+0x7e
wait_init+0x6e
startup+0x201
main+0x1e4
_start+0x83
2 UNPARKED <NONE> 1
libc.so.1`raise+0x2b
libc.so.1`abort+0x10e
utmpx_postfork+0x44
fork_common+0x186
fork_configd+0x8d
fork_configd_thread+0x2ca
libc.so.1`_thrp_setup+0x88
libc.so.1`_lwp_start
=== 0841be7a22d08ad40998470ccadf3bf0 ===
THREAD STATE SOBJ COUNT
1 UNPARKED <NONE> 1
libc.so.1`mutex_lock_impl+0x291
libc.so.1`mutex_lock+0x19
libumem.so.1`umem_cache_alloc+0x5a
libumem.so.1`umem_alloc+0x50
libumem.so.1`malloc+0x36
libuutil.so.1`uu_zalloc+0x1e
libuutil.so.1`uu_list_pool_create+0x7e
wait_init+0x6e
startup+0x201
main+0x1e4
_start+0x83
2 UNPARKED <NONE> 1
libc.so.1`raise+0x2b
libc.so.1`abort+0x10e
utmpx_postfork+0x44
fork_common+0x186
fork_configd+0x8d
fork_configd_thread+0x2ca
libc.so.1`_thrp_setup+0x88
libc.so.1`_lwp_start
=== 93af710f0d808e0cce0a46a4d1bae78b ===
THREAD STATE SOBJ COUNT
1 UNPARKED <NONE> 1
libc.so.1`mutex_lock_impl+0x291
libc.so.1`mutex_lock+0x19
libumem.so.1`vmem_alloc+0x10f
libumem.so.1`umem_slab_create+0x78
libumem.so.1`umem_slab_alloc+0x8a
libumem.so.1`umem_cache_alloc+0x140
libumem.so.1`umem_alloc+0x50
libumem.so.1`malloc+0x36
libuutil.so.1`uu_zalloc+0x1e
libuutil.so.1`uu_list_pool_create+0x7e
wait_init+0x6e
startup+0x201
main+0x1e4
_start+0x83
2 UNPARKED <NONE> 1
libc.so.1`raise+0x2b
libc.so.1`abort+0x10e
utmpx_postfork+0x44
fork_common+0x186
fork_configd+0x8d
fork_configd_thread+0x2ca
libc.so.1`_thrp_setup+0x88
libc.so.1`_lwp_start
...
=== 05298bf53455ffea9b4e85d16a2706ac ===
THREAD STATE SOBJ COUNT
1 UNPARKED <NONE> 1
libc.so.1`mutex_lock_impl+0x291
libc.so.1`mutex_lock+0x19
libumem.so.1`umem_cache_alloc+0x5a
libumem.so.1`umem_alloc+0x50
libumem.so.1`malloc+0x36
libuutil.so.1`uu_zalloc+0x1e
libuutil.so.1`uu_list_pool_create+0x7e
wait_init+0x6e
startup+0x201
main+0x1e4
_start+0x83
2 UNPARKED <NONE> 1
libc.so.1`raise+0x2b
libc.so.1`abort+0x10e
utmpx_postfork+0x44
fork_common+0x186
fork_configd+0x8d
fork_configd_thread+0x2ca
libc.so.1`_thrp_setup+0x88
libc.so.1`_lwp_start
From looking at these stacks, the race becomes clear: fork_configd_thread is racing with startup() – and therefore with utmpx_init(). (That is, utmpx_init() is plowing the lock while we're holding it.) The fix is simple; utmpx_init() must be called before creating the fork_configd() thread.
Updated by
Actions