Bug #9733


Unable to remove stray label leftover from un-exported, removed single-device zfs pool

Added by priyadarshan GD almost 4 years ago. Updated almost 4 years ago.

zfs - Zettabyte File System
Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


On an OmniOS server I imported a single-device zfs pool, used to boot a test FreeBSD install.

After poweroff I removed that disk, forgetting to do proper 'zpool export'.

Now, running 'zpool import' on that OmniOS server gives:

# zpool import
   pool: fido
     id: 7452075738474086658
  state: FAULTED
 status: The pool was last accessed by another system.
 action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.

        fido                     FAULTED  corrupted data
          c1t0025385971B16535d0  UNAVAIL  corrupted data

- Since that disk is not there anymore, I cannot re-import/export the pool properly.

- I also cannot re-attach the disk in order to destroy the pool, since it is not physically in my possession anymore.

- I tried deleting /etc/zfs/zpool.cache and rebooting, with no success, the entry is still there.

- I digged further with the zdb command, which revealed this "label" on disk c1t0025385971B16535d0, which is the server's boot disk:

# zdb -l /dev/rdsk/c1t0025385971B16535d0
failed to unpack label 0
failed to unpack label 1
    version: 5000
    name: 'fido'
    state: 0
    txg: 30770
    pool_guid: 7452075738474086658
    hostid: 647188743
    hostname: ''
    top_guid: 7525102254531229074
    guid: 7525102254531229074
    vdev_children: 1
        type: 'disk'
        id: 0
        guid: 7525102254531229074
        path: '/dev/nvd0p3'
        whole_disk: 1
        metaslab_array: 37
        metaslab_shift: 32
        ashift: 12
        asize: 509746872320
        is_log: 0
        create_txg: 4
failed to unpack label 3

So, zpool import seems to read that stray information on the server boot disk, related to a long-gone pool, and still believes it is available on the system.

- Reading zpool man page I could not find a way to clear that stray entry.

- Now, I am left with option of performing a complete reinstall on that machine, wiping boot disk completely with dd before install.

Is this intendend ZFS behaviour?

Is there a way to safely clear up such a stray zdb entry from a boot disk?

I am not sure if this should be counted as bug or as feature request.

Thanks for considering this.

Actions #1

Updated by Yuri Pankov almost 4 years ago

  • Description updated (diff)
Actions #2

Updated by Yuri Pankov almost 4 years ago

Sorry, re-reading the description, I'm now really confused -- how is the physically gone disk related to any of this? What is the current configuration - rpool on single disk that is c1t0025385971B16535d0 and the import also says that there's a "fido" pool?

Actions #3

Updated by Yuri Pankov almost 4 years ago

Also, /dev/nvd0p3 (device from that "fido" pool label) is an NVMe -- do you have any of those present that are not in the rpool?

Actions #4

Updated by priyadarshan GD almost 4 years ago

Thank you for looking into this.

Yuri Pankov wrote:

Have you tried using zpool labelclear command?

# zpool labelclear -f fido
failed to find device fido, try specifying absolute path instead

Removed pool, "fido", was created on a NVMe disk. It had a bootabe FreeBSD install.

It was attached to the server, imported and exported a few times with success. At one point, the NVMe was removed and it is gone.

The server's rpool is also 1 NVMe disk.

Beside that one, I do not have any other NVMe at the moment. If that would help I could find a new one and install it, but still I would not know how to clear that stray label from rpool.

Actions #5

Updated by priyadarshan GD almost 4 years ago

I have been trying to replicate this, by having drives imported, powering off, removing them without exporting them first, and see if zpool import was giving same message, but with no success.

I believe the issue happened when I booted from "fido" zpool (the NVMe disk with FreeBSD OS). Although I did not import rpool (I was just testing), could it be FreeBSD somehow wrote that label on the server rpool? That would make it more a FreeBSD bug that illumos'.

If that’s true, I’d be wary of importing FreeBSD pools on OmniOS servers, since they "pollute" the rpool with foreign data difficult or impossible to purge, if something does not go as expected.

Anyway, I have ordered a new NVMe, it should come in next few days. If you have time, would you let me know steps you would advice in order to avoid wiping out rpool and re-intalling OmniOS?

Thank you for the help.

Actions #6

Updated by priyadarshan GD almost 4 years ago

I got a secoond NVMe. I completely reinstalled OmniOS, and I was able to replicate the issue with these steps:

- Install a second NVMe drive
- Install FreeBSD on that and boot from it
- Boot again, this time from OmniOS
- Import the FreeBSD pool (on second NVMe)
- Poweroff without exporting it
- Remove FreeBSD NVMe from server
- Boot from OmniOS

At that point, zpool import will show message as original message.

In that state, there is nothing one can do to remove the stray label from boot disk.

Although not a real bug, may I submit this as feature request, i.e., to be able to remove stray leftover labels caused by above steps?

Or, if that is not applcable, please feel free to close this ticket. I learned that FreeBSD-created pools (and ZOL's) do not mix well with illumos.

Thanks to all for the help and advice.


Also available in: Atom PDF