illumos gate

Several times, the Jenkins instance I used to provide putback-triggered
illumos builds and packages has inexplicably hung while polling for new SCM
changes. A message is produced saying that no polling activity is occuring,
and builds cease.

I've, twice, investigated why and found the following things to be true.

• Jenkins has forked and has a child process
• This child has not called exec (though presumably wishes it had)
• This child has existed since approximately the last SCM poll + one interval

The parent process is still, unfortunately, live, so its state as compared to
the child is represented only by the child, to the extent the child has not
modified it post-fork.

The state of the child looks like this:

b38e8008 libc_hwcap1.so.1`__lwp_park+0x19(b5154a40, 0, b5eca148, 0)
b38e8048 libc_hwcap1.so.1`mutex_lock_impl+0x163(b5eca148, 0, b5e08143, b5ea0018, b5eb0e30, 0)
b38e8068 libc_hwcap1.so.1`mutex_lock+0x10(b5eca148)
b38e8088 libcryptoutil.so.1`pkcs11_close_common+0x1c(b5eca110, b5eca148, 1, b5eb677c)
b38e80a8 libcryptoutil.so.1`pkcs11_close_urandom+0x29(0, b5e74000, b38e80d8, b5e7440c, b5e74420, 0)
b38e80d8 pkcs11_softtoken.so.1`finalize_common+0xaa(1)
b38e80f8 pkcs11_softtoken.so.1`softtoken_fini+0x3b(feffb8f4, 0, b5ea0718, b38e814c, fefcdbdf, 0)
b38e810c pkcs11_softtoken.so.1`_fini+0x25(0, 0, feffb388, feffb2e4, feffb168, b5f10810)
b38e814c ld.so.1`call_fini+0xff(feffb168, b4e60768, b5f10810, 0)
b38e81bc ld.so.1`remove_hdl+0x78c(b5ea0bc8, b5f10810, 0, fefd265a)
b38e81dc ld.so.1`dlclose_core+0xbf(b5ea0bc8, b5f10810, feffb168, fefd2756)
b38e821c ld.so.1`dlclose_intn+0x2a(b5ea0bc8, b5f10810, 0, fefd27c8)
b38e825c ld.so.1`dlclose_check+0x97(b5ea0bc8, b5f10810, 0, fefd288d)
b38e829c ld.so.1`dlclose+0x46(b5ea0bc8, b5efa000, b38e82cc, b5ee5bc2)
b38e82dc libpkcs11.so.1`pkcs11_slottable_delete+0xbf(fedc07c0, b5efa000, b38e830c, b5ee182e, 0, b5e74000)
b38e82ec libpkcs11.so.1`finalize_common+0xe2(0)
b38e830c libpkcs11.so.1`pkcs11_fini+0x46(fedc0780, fedc0080, b38e832c, fee2320d, 0, fef20000)
b38e831c libpkcs11.so.1`pkcs11_fork_child_fini+0xb(0, fef20000, b38e835c, fee8c791, fef20000, fef24580)
b38e832c libc_hwcap1.so.1`_postfork_child_handler+0x2f(fef20000, fef24580, fef24540, fef244c0, 0, 93fcd18)
b38e835c libc_hwcap1.so.1`forkx+0x199(0, 0, fee3a642, fee8c844, fe262000)
b38e837c libc_hwcap1.so.1`fork+0x1d(93fcc00, b87a61a8, b87a61b0, e6a0ce24, 15, e6a0cd8c)
b38e83e8 libjava.so`Java_java_lang_UNIXProcess_forkAndExec+0x395(93fcd18, 
b38e8480, b38e847c, b38e8478, 7, b38e8470)
b38e8448 0xfb00a152(0, fb0080f9, e6a0ce70, e6a0ce60, e6a0ce50, 1)
b38e84a8 0xfb002f69(1, e6a0ce18, f, e6a0ca48, 7, e6a0c978)
e6a0ccd0 0xfb06b7f8(0, 0, 0, e6a0ce50, e6a0ce60, e6a0ce70)
00000105 0xb87a6f30()

We just forked, and we're running the atfork handles for libpkcs11, one of
which deletes the slottable, which dlclose's the subordinate modules,
including pkcs11_softtoken, that closes the urandom device via
pkcs11_close_urandom, and tries to take the libcryptoutil &urandom_mutex,
which is already held.

It is already held because, at the moment the parent (main jenkins) process
forked it held &urandom_mutex, and thus it is also held in the child process.
However, the child does not contain the thread which took the mutex, so it
will never be released, and the child is deadlocked under the (child's side of)
fork(2).