Cannot Fix pool misconfiguration because other top level device is raidz.
I had a slight mishap. Instead of typing
zpool add bkpool spare c18t0d0
zpool add bkpool c18t0d0
without the spare. So basicly added the device as top level vdev to the pool.
And it turns out I now cannot remove that device:
zpool remove bkpool c18t0d0
cannot remove c18t0d0: invalid config; all top-level vdevs must have the same sector size and not be raidz.
That is a problem on my part and I probybly am screwed now as this is a productive pool. And I have absolutely no chance of migrating to another pool anytime soon.
However I am of the opinion that the removal should be allowed to work given that the disk Is only single and not raidz itself and there is no section in zpool-features(5) mentioning that problem.
Updated by Till Wegmüller 11 months ago
Turns out there was either a regression or a missing Patch back on OI 2018.04 that allowed me to add the device without requiring -f
Server Osnet-incorporation is 2018.0.0-17625
In my Local Machines OI 2018.10 I needed to use -f to be able to test the situation. osnet-incorporation 2018.0.0-17649
Updated by Till Wegmüller 2 months ago
After some testing with the Machine I found the Problem.
Pre-flight Checks in ZFS react differently than ZFS when it is writing to disk. Basically the Pre-flight checks deduplicates disk that have the same serial. This happens with VIRTIOBLK unless you have the serial option set on the Qemu Command line. So this problem is caused by that behavior and me not having the serial option set before. I now have fixed that with new disks.