Project

General

Profile

Bug #3987

svc.startd dies in utmpx_postfork()

Added by Robert Mustacchi over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Category:
cmd - userland programs
Start date:
2013-08-04
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

A thoth analyzer was written to look at threads in the process:

#
# This analyzer only applies to core files
#
if [ "$THOTH_TYPE" != "core" ]; then
    exit 0;
fi

#
# This is only OS-2359 if we have utmpx_postfork in our stack
#
if ( ! echo ::stack | mdb $THOTH_DUMP | grep utmpx_postfork > /dev/null ); then
    exit 0;
fi

#
# We're interested in this dump; emit all stacks
#
echo ::stacks | mdb $THOTH_DUMP

Here's the output of that:

$ thoth analyzer OS-2359-stacks < OS-2359-stacks
thoth: reading analyzer 'OS-2359-stacks' from stdin
thoth: added analyzer 'OS-2359-stacks'
$ thoth analyze cmd=svc.startd OS-2359-stacks
=== 40214dd1db0d0cbe03b8c1b07ed5caf1 ===

THREAD   STATE    SOBJ        COUNT
1        UNPARKED <NONE>          1
         libc.so.1`mutex_lock_impl+0x291
         libc.so.1`mutex_lock+0x19
         libumem.so.1`vmem_alloc+0x10f
         libumem.so.1`umem_slab_create+0x78
         libumem.so.1`umem_slab_alloc+0x8a
         libumem.so.1`umem_cache_alloc+0x140
         libumem.so.1`umem_alloc+0x50
         libumem.so.1`malloc+0x36
         libuutil.so.1`uu_zalloc+0x1e
         libuutil.so.1`uu_list_pool_create+0x7e
         wait_init+0x6e
         startup+0x201
         main+0x1e4
         _start+0x83

2        UNPARKED <NONE>          1
         libc.so.1`raise+0x2b
         libc.so.1`abort+0x10e
         utmpx_postfork+0x44
         fork_common+0x186
         fork_configd+0x8d
         fork_configd_thread+0x2ca
         libc.so.1`_thrp_setup+0x88
         libc.so.1`_lwp_start

=== ef4b91b304ee21607dae77a7a5f770b8 ===

THREAD   STATE    SOBJ        COUNT
2        UNPARKED <NONE>          1
         libc.so.1`raise+0x2b
         libc.so.1`abort+0x10e
         utmpx_postfork+0x44
         fork_common+0x186
         fork_configd+0x8d
         fork_configd_thread+0x2ca
         libc.so.1`_thrp_setup+0x88
         libc.so.1`_lwp_start

=== e7d8f040512051792368d6bf1a4389b4 ===

THREAD   STATE    SOBJ        COUNT
1        UNPARKED <NONE>          1
         libc.so.1`mutex_lock_impl+0x291
         libc.so.1`mutex_lock+0x19
         libumem.so.1`umem_cache_alloc+0x5a
         libumem.so.1`umem_alloc+0x50
         libumem.so.1`malloc+0x36
         libuutil.so.1`uu_zalloc+0x1e
         libuutil.so.1`uu_list_pool_create+0x7e
         wait_init+0x6e
         startup+0x201
         main+0x1e4
         _start+0x83

2        UNPARKED <NONE>          1
         libc.so.1`raise+0x2b
         libc.so.1`abort+0x10e
         utmpx_postfork+0x44
         fork_common+0x186
         fork_configd+0x8d
         fork_configd_thread+0x2ca
         libc.so.1`_thrp_setup+0x88
         libc.so.1`_lwp_start

=== 000c3a37469868df1eb1c74f60ebbf25 ===

THREAD   STATE    SOBJ        COUNT
1        UNPARKED <NONE>          1
         libc.so.1`mutex_lock_impl+0x291
         libc.so.1`mutex_lock+0x19
         libumem.so.1`vmem_alloc+0x10f
         libumem.so.1`umem_slab_create+0x78
         libumem.so.1`umem_slab_alloc+0x8a
         libumem.so.1`umem_cache_alloc+0x140
         libumem.so.1`umem_alloc+0x50
         libumem.so.1`malloc+0x36
         libuutil.so.1`uu_zalloc+0x1e
         libuutil.so.1`uu_list_pool_create+0x7e
         wait_init+0x6e
         startup+0x201
         main+0x1e4
         _start+0x83

2        UNPARKED <NONE>          1
         libc.so.1`raise+0x2b
         libc.so.1`abort+0x10e
         utmpx_postfork+0x44
         fork_common+0x186
         fork_configd+0x8d
         fork_configd_thread+0x2ca
         libc.so.1`_thrp_setup+0x88
         libc.so.1`_lwp_start

=== e40609350344472fce251e9d6c10440c ===

THREAD   STATE    SOBJ        COUNT
1        UNPARKED <NONE>          1
         libc.so.1`mutex_lock_impl+0x291
         libc.so.1`mutex_lock+0x19
         libumem.so.1`vmem_alloc+0x10f
         libumem.so.1`umem_slab_create+0x78
         libumem.so.1`umem_slab_alloc+0x8a
         libumem.so.1`umem_cache_alloc+0x140
         libumem.so.1`umem_alloc+0x50
         libumem.so.1`malloc+0x36
         libuutil.so.1`uu_zalloc+0x1e
         libuutil.so.1`uu_list_pool_create+0x7e
         wait_init+0x6e
         startup+0x201
         main+0x1e4
         _start+0x83

2        UNPARKED <NONE>          1
         libc.so.1`raise+0x2b
         libc.so.1`abort+0x10e
         utmpx_postfork+0x44
         fork_common+0x186
         fork_configd+0x8d
         fork_configd_thread+0x2ca
         libc.so.1`_thrp_setup+0x88
         libc.so.1`_lwp_start

=== 0841be7a22d08ad40998470ccadf3bf0 ===

THREAD   STATE    SOBJ        COUNT
1        UNPARKED <NONE>          1
         libc.so.1`mutex_lock_impl+0x291
         libc.so.1`mutex_lock+0x19
         libumem.so.1`umem_cache_alloc+0x5a
         libumem.so.1`umem_alloc+0x50
         libumem.so.1`malloc+0x36
         libuutil.so.1`uu_zalloc+0x1e
         libuutil.so.1`uu_list_pool_create+0x7e
         wait_init+0x6e
         startup+0x201
         main+0x1e4
         _start+0x83

2        UNPARKED <NONE>          1
         libc.so.1`raise+0x2b
         libc.so.1`abort+0x10e
         utmpx_postfork+0x44
         fork_common+0x186
         fork_configd+0x8d
         fork_configd_thread+0x2ca
         libc.so.1`_thrp_setup+0x88
         libc.so.1`_lwp_start

=== 93af710f0d808e0cce0a46a4d1bae78b ===

THREAD   STATE    SOBJ        COUNT
1        UNPARKED <NONE>          1
         libc.so.1`mutex_lock_impl+0x291
         libc.so.1`mutex_lock+0x19
         libumem.so.1`vmem_alloc+0x10f
         libumem.so.1`umem_slab_create+0x78
         libumem.so.1`umem_slab_alloc+0x8a
         libumem.so.1`umem_cache_alloc+0x140
         libumem.so.1`umem_alloc+0x50
         libumem.so.1`malloc+0x36
         libuutil.so.1`uu_zalloc+0x1e
         libuutil.so.1`uu_list_pool_create+0x7e
         wait_init+0x6e
         startup+0x201
         main+0x1e4
         _start+0x83

2        UNPARKED <NONE>          1
         libc.so.1`raise+0x2b
         libc.so.1`abort+0x10e
         utmpx_postfork+0x44
         fork_common+0x186
         fork_configd+0x8d
         fork_configd_thread+0x2ca
         libc.so.1`_thrp_setup+0x88
         libc.so.1`_lwp_start

...

=== 05298bf53455ffea9b4e85d16a2706ac ===

THREAD   STATE    SOBJ        COUNT
1        UNPARKED <NONE>          1
         libc.so.1`mutex_lock_impl+0x291
         libc.so.1`mutex_lock+0x19
         libumem.so.1`umem_cache_alloc+0x5a
         libumem.so.1`umem_alloc+0x50
         libumem.so.1`malloc+0x36
         libuutil.so.1`uu_zalloc+0x1e
         libuutil.so.1`uu_list_pool_create+0x7e
         wait_init+0x6e
         startup+0x201
         main+0x1e4
         _start+0x83

2        UNPARKED <NONE>          1
         libc.so.1`raise+0x2b
         libc.so.1`abort+0x10e
         utmpx_postfork+0x44
         fork_common+0x186
         fork_configd+0x8d
         fork_configd_thread+0x2ca
         libc.so.1`_thrp_setup+0x88
         libc.so.1`_lwp_start

From looking at these stacks, the race becomes clear: fork_configd_thread is racing with startup() – and therefore with utmpx_init(). (That is, utmpx_init() is plowing the lock while we're holding it.) The fix is simple; utmpx_init() must be called before creating the fork_configd() thread.

History

#1

Updated by Robert Mustacchi over 6 years ago

  • Status changed from New to Resolved

Resolved in 0d421f668cdfd7a53019f57234af254738038aa0.

Also available in: Atom PDF