illumos gate

In the init function of libnsl(3lib) called _libnsl_lock_init() there is an attempt to be fork(2) safe and the pthread_atfork(3c) is called.

Unfortunately, the current implementation is not sufficient. The problem could be reproduced using the attached tli-fork.c test in this way:

$ gcc -Wall -o tli-fork -lnsl tli-fork.c 
$ ./tli-fork 3 5
finishing...
^Z
[1]+  Stopped                 ./tli-fork 3 5
$ pgrep -x tli-fork | sort -n | tail -1
831728
$ pstack 831728
831728:    ./tli-fork 3 5
 feee6199 lwp_park (0, 0, 0)
 feedf048 mutex_lock_internal (8061a94, 0, 1, feb4543a, febaa00c, fee00080) + 3b8
 feedf3be mutex_lock_impl (8061a94, 0, fe930f38, fee5d8a9, feebdaab) + 2aa
 feedfd91 mutex_lock (8061a94, 0, 0, feb4545a) + 64
 feb45465 sig_mutex_lock (8061a94, 0, 2, 0, 0, fef5e240) + 23
 feb5e6b9 _tx_look (3, 2, 0, fe930fb8) + 42
 feb6302d _xti_look (3, 1, 8, fef5e1c0, feebff09, 3) + 1d
 080513c5 forker   (0, 0, 0, 0) + 85
 feee5fad _thrp_setup (fecb1a40) + 88
 feee6140 _lwp_start (fecb1a40, 0, 0, 0, 0, 0)
$

The first parameter of tli-fork is the number of threads (more threads, easier to reproduce). The second parameter is time (in seconds) we want to run the test.

As we see, the forker() hung in t_look() because some other thread is holding the mutex, but such thread was not replicated on fork(2), so there is nobody who would unlock the mutex in the child.

To solve this issue we need to acquire all mutexes (including all ti_lock mutexes from struct _ti_user) and rwlocks in _libnsl_prefork().