illumos gate

Illumos 4080¹ allows "zpool clear" to work on readonly pools: i don't think this is the intended behaviour, we shouldn't be allowed to clear readonly pools. Probably.

Additionally, the following panic is reproducible trying to zpool clear a readonly pool:

> ::status
debugging crash dump vmcore.1 (64-bit) from openindiana
operating system: 5.11 master-0-g8dcfe5e5a1 (i86pc)
image uuid: 49f26dc4-a94a-6177-c9d4-b1ca68f101e3
panic message: assertion failed: spa_writeable(spa), file: ../../common/fs/zfs/vdev.c, line: 3110
dump content: kernel pages only
> ::stack
vpanic()
0xfffffffffbdfed08()
vdev_state_dirty+0x100(ffffff04e130ac80)
vdev_clear+0x1d8(ffffff04e19fa000, ffffff04e1309f80)
vdev_clear+0xa6(ffffff04e19fa000, ffffff04e130ac80)
vdev_clear+0xa6(ffffff04e19fa000, 0)
zfs_ioc_clear+0x11a(ffffff04e1c50000)
zfsdev_ioctl+0x50f(10e00000000, 5a21, 8042b48, 100003, ffffff04c6b7b0e8, ffffff001436ae58)
cdev_ioctl+0x39(10e00000000, 5a21, 8042b48, 100003, ffffff04c6b7b0e8, ffffff001436ae58)
spec_ioctl+0x60(ffffff04c44e5800, 5a21, 8042b48, 100003, ffffff04c6b7b0e8, ffffff001436ae58)
fop_ioctl+0x55(ffffff04c44e5800, 5a21, 8042b48, 100003, ffffff04c6b7b0e8, ffffff001436ae58)
ioctl+0x9b(3, 5a21, 8042b48)
_sys_sysenter_post_swapgs+0x237()
> ffffff04e130ac80::print vdev_t vdev_spa->spa_mode
vdev_spa->spa_mode = 0x1
> ffffff04e130ac80::print vdev_t vdev_cant_write   
vdev_cant_write = 0 (0)
> 

It's not clear to me how we managed to have a readonly SPA with, at least, one writable VDEV. This issue cannot be reproduced on single-device and stripe pools, only mirror and raidz are affected. The writable device seems to be a non-leaf VDEV in every failure.

A fix² is already in the ZFS on Linux repository to addess this issue

[1] https://www.illumos.org/issues/4080 "zpool clear fails to clear pool"
[2] https://github.com/zfsonlinux/zfs/commit/92e43c17188d47f47b69318e4884096dec380e36