#14022 added a DKIOSTATE ioctl to the zpool online -e path, which appears now to reliably hang at least on AWS systems using the xdf driver. Looking at a stuck zpool online -e process:

# pargs 548
548:    zpool online -e rpool c1t0d0
argv[0]: zpool
argv[1]: online
argv[2]: -e
argv[3]: rpool
argv[4]: c1t0d0

# mdb -p 548
Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libtopo.so.1 libavl.so.1 libnvpair.so.1 ]
> $C
fffffc7fffdf2ca0 libc.so.1`ioctl+0xa()
fffffc7fffdf4310 libzfs.so.1`zpool_vdev_online+0x2d0(c3bdd0, fffffc7fffdf864e, 8, fffffc7fffdf4334)
fffffc7fffdf4380 zpool_do_online+0x135(4, fffffc7fffdf8440)
fffffc7fffdf83e0 main+0xdf(5, fffffc7fffdf8438)
fffffc7fffdf8410 _start_crt+0x87()
fffffc7fffdf8420 _start+0x18()
> libzfs.so.1`zpool_vdev_online+0x2d0::dis -n 1
libzfs.so.1`zpool_vdev_online+0x2cb:    call   -0x460   <libzfs.so.1`zpool_relabel_disk>
libzfs.so.1`zpool_vdev_online+0x2d0:    jmp    -0x18e   <libzfs.so.1`zpool_vdev_online+0x147>
libzfs.so.1`zpool_vdev_online+0x2d5:    nopl   (%rax)

In the kernel:

> 0t548::pid2proc | ::walk thread | ::findstack -v
stack pointer for thread fffffe0389a2e060 (zpool/1): fffffe0006af8af0
[ fffffe0006af8af0 _resume_from_idle+0x12b() ]
  fffffe0006af8b20 swtch+0x133()
  fffffe0006af8b90 cv_wait_sig+0x171(fffffe03860feef8, fffffe03860fee40)
  fffffe0006af8bd0 xdf_dkstate+0x73(fffffe03860feb80, 0)
  fffffe0006af8cc0 xdf_ioctl+0x4f6(f700000000, 40d, fffffc7fffdf244c, 202007, fffffe0390bcf970, fffffe0006af8e18)
  fffffe0006af8d00 cdev_ioctl+0x2b(f700000000, 40d, fffffc7fffdf244c, 202007, fffffe0390bcf970, fffffe0006af8e18)
  fffffe0006af8d50 spec_ioctl+0x45(fffffe0391d45e40, 40d, fffffc7fffdf244c, 202007, fffffe0390bcf970, fffffe0006af8e18, 0)
  fffffe0006af8de0 fop_ioctl+0x5b(fffffe0391d45e40, 40d, fffffc7fffdf244c, 202007, fffffe0390bcf970, fffffe0006af8e18, 0)
  fffffe0006af8f00 ioctl+0x153(7, 40d, fffffc7fffdf244c)
  fffffe0006af8f10 sys_syscall+0x1a8()

> ::prtconf -d xdf | ::devinfo -d | ::print xdf_t xdf_state xdf_xdev_nblocks xdf_xdev_secsize xdf_mstate xdf_dinfo
xdf_state = 3 (XD_READY)
xdf_xdev_nblocks = 0x6400000
xdf_xdev_secsize = 0x200
xdf_mstate = 0 (DKIO_NONE)
xdf_dinfo = 0

It would seem that, for a non-removal drive (xdf_dinfo has neither VDISK_CDROM nor VDISK_REMOVABLE) the xdf_mstate value remains DKIO_NONE forever.

It is not clear to me, at least from reading dkio(4I), what DKIOCSTATE is supposed to mean for drives that are not removable:

     DKIOCSTATE

        This ioctl(2) blocks until the state of the drive, inserted or
        ejected, is changed.  The argument is a pointer to a dkio_state, enum,
        whose possible enumerations are listed below.  The initial value
        should be either the last reported state of the drive, or DKIO_NONE.
        Upon return, the enum pointed to by the argument is updated with the
        current state of the drive.

          enum dkio_state {
              DKIO_NONE,          /* Return disk's current state */
              DKIO_EJECTED,       /* Disk state is 'ejected' */
              DKIO_INSERTED       /* Disk state is 'inserted' */
          };

Note that this documentation is missing the DKIO_DEV_GONE variant which appears in uts/common/sys/dkio.h. In that header, DKIOCSTATE is explicitly mentioned as being for removable devices:

/*
 * The following ioctl's are removable media support
 */
#define DKIOCLOCK       (DKIOC|7)       /* Generic 'lock' */
#define DKIOCUNLOCK     (DKIOC|8)       /* Generic 'unlock' */
#define DKIOCSTATE      (DKIOC|13)      /* Inquire insert/eject state */
#define DKIOCREMOVABLE  (DKIOC|16)      /* is media removable */

It seems that a common pattern for disk drivers is initialising their insertion state to DKIO_NONE, then shortly afterwards changing to DKIO_INSERTED, at least for some types of media. This is what blkdev does (and thus nvme and vioblk), and roughly what sd does as well. In other cases:

• lofi is willing to return DKIO_INSERTED or DKIO_DEV_GONE depending on whether the device has been forcefully unmapped since having been installed, and appears to correctly block waiting for change, and correctly handle DKIO_NONE
• ramdisk always returns DKIO_INSERTED no matter what is going on, which does not seem to be correct -- it should block forever if one passes in DKIO_INSERTED
• xdf, at issue here, does not correctly handle returning the current state on DKIO_NONE, at least on account of never moving away from that state for non-removable drives
• cmdk (IDE) seems like it will just fail a DKIOCSTATE request with ENXIO if the drive is not removable (dad_rmb), but if the drive is removable and does not report no media, it seems like ata_disk_state() might leave it with DKIO_NONE as well
• zvols probably always fail with ENOTTY

Suffice it to say, outside of blkdev, implementations of DKIOCSTATE are a mixed bag and I'm not even sure exactly how it is intended to work -- should it fail when things are not removable, or should it always pretend that the disk is DKIO_INSERTED? Is there not a more explicit way we can refresh the label if that's what is needed for #14022 ?