illumos gate

I've recently been given a syseventd core dump with the following stack:

feb8fc08 libc.so.1`_lwp_kill+0x15(4, 6, feb8fc38, fef65000, fef65000, feb8fca0)
feb8fc28 libc.so.1`raise+0x2b(6, 0, feb8fc40, feeb9890, 0, 0)
feb8fc78 libc.so.1`abort+0x10e(feb8fca0, feb8fca0, 83, fdc53dcc, fdc53d08, 58)
feb8fea8 0xfee6c597(fdc53dcc, fdc53d08, 58, fdc53853)
feb8fed8 libcmdutils.so.1`list_destroy+0x57(fe8741a0)
feb8ff08 zfs_mod.so`slm_fini+0x9f(2)
feb8ff48 unload_modules+0xc4(1, 805520c, feb8ff74, feb8ffac, feb8ffbc, 0)
feb8ff68 syseventd_fini+0x41(1, 1000000, 0, feb8ffac, feb8ffbc, 0)
feb8ff88 hup_handler+0x2f(1, feb8ffbc, 0, 0)
feb8ffc8 sigwait_thr+0x55(0, 0, 0, 0)
feb8ffe8 libc.so.1`_thrp_setup+0x88(fec91240)
feb8fff8 libc.so.1`_lwp_start(fec91240, 0, 0, 0, 0, 0)

The abort is caused by an assertion in list_destroy(), where it checks that the list is empty, which it isn't in this case. I conclude that there is a race against the thread used to enumerate the pools (zfs_enum_pools(), created by slm_init()). Apparently that thread is still active and adding pools to the list while we're already in slm_fini() removing entries from the list. When slm_fini() finished removing all entries it waits for the thread to complete, which can still add entries to the list while slm_fini() waits. Then slm_fini() destroys the list and run into the assertion.

I think the fix for this is to move the thr_join() call all the way up to the beginning of the function..