illumos gate

ZFS uses ldi_open_by_devid(9F), et al, to gain access to block devices when importing a pool. This bumps the devi_ref member of the opened device node, so that it doesn't become unreferenced (and thus freed) while we have it open. Unfortunately, very little will cause ZFS to actually close its LDI handle, and leave it closed; least of all with timing adequate to effect device removal.

As it turns out, LDI provides a basic facility for notifying the owner of an open handle that the device has been offlined (or otherwise faulted, removed, etc). This should really not be considered optional, as correct operation of device removal seems impossible without it. Regardless, ZFS needs to listen for these events and close its LDI handle at appropriate moments.

In addition to closing the device handle, we should also update the state of the vdev that was using the device to be REMOVED.