illumos gate

Andy Fiddaman wrote:

Attempting to expand a ZFS pool in one operation using zpool online -e does not work, causing pool access (and the machine) to hang.
If the partition is expanded first using format, however, then it succeeds.

EFI wholedisk or custom partitioning?

Toomas Soome wrote in #note-1:

EFI wholedisk or custom partitioning?

I've expanded the description - the reproducer I'm using is EFI whole disk, so that zpool online -e tries to expand it

The pool ends up being suspended because ldi_get_size() returns -1 - called from vdev_disk_open() in the re-open case

dtrace -n 'fbt::ldi_get_size:entry{v = (struct ldi_handle *)arg0; print(*v); self->t++}' -n 'fbt::ldi*:entry/self->t/{}' -n 'fbt::ldi*:return/self->t/{trace(arg1)}' -F
CPU FUNCTION
  2  -> ldi_get_size                          struct ldi_handle {
    struct ldi_handle *lh_next = 0
    uint_t lh_ref = 0x1
    uint_t lh_flags = 0
    uint_t lh_type = 0x2
    struct ldi_ident *lh_ident = 0xfffffe064a4abbf8
    vnode_t *lh_vp = 0xfffffe064a4c3a40
    kmutex_t [1] lh_lock = [
        kmutex_t {
            void *[1] _opaque = [ 0 ]
        }
    ]
    struct ldi_event *lh_events = 0
}
  2   | ldi_get_size:entry
  2    -> ldi_get_otyp
  2    <- ldi_get_otyp                                        0
  2    -> ldi_prop_exists
  2    <- ldi_prop_exists                                     0
  2    -> ldi_prop_exists
  2    <- ldi_prop_exists                                     0
  2    -> ldi_prop_exists
  2    <- ldi_prop_exists                                     0
  2    -> ldi_prop_exists
  2    <- ldi_prop_exists                                     0
  2  <- ldi_get_size                                 4294967295

What is happening here is that during an online expansion of an EFI disk libzfs:zpool_vdev_online() is calling libzfs:zpool_relabel_disk() which is turn calling libefi:efi_use_whole_disk(). efi_use_whole_disk() uses the DKIOCSETEFI ioctl to write the new GPT table header, which ends up calling cmlb_dkio_set_efi(). After writing the table, this function sets cl_f_geometry_is_valid = B_FALSE; on the LUN.

When libzfs then continues and tries to re-open the device, there is a race where it can attempt to read the device size while the geometry is still marked as invalid. Adding a small pause here makes the resize work most of the time.

  0    -> vdev_reopen
  0      -> vdev_close
  0      <- vdev_close                                        1
  0      -> vdev_open
  0        -> vdev_disk_open
  0          -> ldi_get_size
  0            -> ldi_prop_exists
  0              -> cmlb_prop_op
  0                -> cmlb_is_valid
  0                <- cmlb_is_valid                           0           <- due to cl_f_geometry_is_valid being false
  0              <- cmlb_prop_op                              1
  0            <- ldi_prop_exists                             0
  0          <- ldi_get_size                         4294967295
  0           | zfs_dbgmsg:zfs-dbgmsg         disk vdev '/dev/dsk/c1t0d0s1': vdev_disk_open: failed to get size
  0        <- vdev_disk_open                                 22
  0        -> vdev_set_state
  0          -> vdev_is_dead

Perhaps rather than a small pause we could adjust things so that we can detect this condition explicitly and retry until it resolves?

The small pause was just a diagnostic (dtrace chill) to try and isolate the problem. I have a proposed fix that is at least in the right area.

I've tested the patch in the associated gerrit review on an OmniOS VM running under bhyve. Prior to this the pool faulted 100% of the time in a run of 20 attempts, and afterwards I had 20 successful consecutive runs. Here is some output from one of them:

root@test:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  7.50G  2.15G  5.35G        -        3G     2%    28%  1.00x  ONLINE  -
root@test:~# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c1t0d0    ONLINE       0     0     0

errors: No known data errors
root@test:~# prtvtoc -s /dev/rdsk/c1t0d0
*                            First       Sector      Last
* Partition  Tag  Flags      Sector       Count      Sector  Mount Directory
       0     12    00          256      524288      524543
       1      4    00       524544    16236255    16760798
       8     11    00     16760799       16384    16777182

root@test:~# zpool online -e rpool c1t0d0

root@test:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  10.5G  2.15G  8.35G        -         -     1%    20%  1.00x  ONLINE  -

root@test:~# prtvtoc -s /dev/rdsk/c1t0d0
*                            First       Sector      Last
* Partition  Tag  Flags      Sector       Count      Sector  Mount Directory
       0     12    00          256      524288      524543
       1      4    00       524544    22527711    23052254
       8     11    00     23052255       16384    23068638

Using dtrace, I can see that the geometry is being revalidated as a result of the new ioctl() call:

   0  47448          cmlb_dkio_set_efi:entry
  0  47522                 cmlb_ioctl:entry   DKIOCSTATE
  0  47418     cmlb_validate_geometry:entry
              cmlb`cmlb_ioctl+0x9d
              blkdev`bd_ioctl+0xc5
              genunix`cdev_ioctl+0x3f
              specfs`spec_ioctl+0x55
              genunix`fop_ioctl+0x40
              genunix`ioctl+0x144
              unix`sys_syscall+0x2ae

  0   1572            zfs_dbgmsg:zfs-dbgmsg spa=rpool async request task=1

This is the script I used here:

#!/usr/sbin/dtrace -Cs

zfs-dbgmsg{printf("%s\n", stringof(arg0))}

fbt::cmlb_ioctl:entry/arg2 == 0x40d/ { self->t++; trace("DKIOCSTATE") }
fbt::cmlb_ioctl:return/self->t/ { self->t--; }
fbt::cmlb_validate_geometry:entry/self->t/ { stack() }
fbt::cmlb_dkio_set_efi:entry {}

For completeness, I also checked that there is no regression around legacy pools on MBR-partitioned drives:

omnios@bhyvetest:~$ zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        rpool                      ONLINE       0     0     0
          c1t589CFC2045C20001d0s0  ONLINE       0     0     0

errors: No known data errors
omnios@bhyvetest:~$ pfexec fdisk -W - c1t589CFC2045C20001d0p0 | tail -5
* Id    Act  Bhead  Bsect  Bcyl    Ehead  Esect  Ecyl    Rsect      Numsect
  191   128  65     2      0       255    63     1023    4096       16773120
  0     0    0      0      0       0      0      0       0          0
  0     0    0      0      0       0      0      0       0          0
  0     0    0      0      0       0      0      0       0          0

omnios@bhyvetest:~$ zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  7.50G   767M  6.75G        -         -     0%     9%  1.00x  ONLINE  -

omnios@bhyvetest:~$ pfexec /usr/lib/sysevent/sysevent -r
nvlist version: 0
        date = 2022-02-18T09:54:22.558Z
        type = ready

... at this point, I resized the underlying ZVOL from outside the VM and checked that the sysevent was
... received.

nvlist version: 0
        date = 2022-02-18T09:54:52.108Z
        type = event
        vendor = SUNW
        publisher = nvme
        class = EC_dev_status
        subclass = ESC_dev_dle
        pid = 0
        data = (embedded nvlist)
        nvlist version: 0
                phys_path = /devices/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC2045C20001,0:a
        (end data)

.. and, as expected, no change on the pool

root@bhyvetest:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  7.50G   767M  6.75G        -         -     0%     9%  1.00x  ONLINE  -

root@bhyvetest:~# zpool online -e rpool c1t589CFC2045C20001d0s0

root@bhyvetest:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  7.50G   767M  6.75G        -         -     0%     9%  1.00x  ONLINE  -

and here is a test with a pool on a GPT-partitioned disk, first without a UEFI boot partition:

root@bhyvetest:~# zpool create bob c2t0d0
root@bhyvetest:~# prtvtoc -s /dev/rdsk/c2t0d0p0
*                            First       Sector      Last
* Partition  Tag  Flags      Sector       Count      Sector  Mount Directory
       0      4    00          256    20954847    20955102
       8     11    00     20955103       16384    20971486
root@bhyvetest:~# zpool list bob
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
bob   9.50G  1.12M  9.50G        -         -     0%     0%  1.00x  ONLINE  -
root@bhyvetest:~# zdb | grep phys_path
            phys_path: '/pci@0,0/pci1af4,2@5/blkdev@0,0:a'          <-- note 'a'

... volume resize, outside the VM

root@bhyvetest:~# zpool list bob
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
bob   9.50G  1.12M  9.50G        -       11G     0%     0%  1.00x  ONLINE  -
root@bhyvetest:~# zpool online -e bob c2t0d0
root@bhyvetest:~# zpool list bob
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
bob   20.5G  1.38M  20.5G        -         -     0%     0%  1.00x  ONLINE  -

and with a UEFI boot partition:

root@bhyvetest:~# zpool create -B bob c2t0d0
root@bhyvetest:~# prtvtoc -s /dev/rdsk/c2t0d0p0
*                            First       Sector      Last
* Partition  Tag  Flags      Sector       Count      Sector  Mount Directory
       0     12    00          256      524288      524543
       1      4    00       524544    43499231    44023774
       8     11    00     44023775       16384    44040158
root@bhyvetest:~# zpool list bob
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
bob   20.5G   744K  20.5G        -        1G     0%     0%  1.00x  ONLINE  -
root@bhyvetest:~# zdb | grep phys_path
            phys_path: '/pci@0,0/pci1af4,2@5/blkdev@0,0:b'          <-- note 'b'

... volume resize, outside the VM

root@bhyvetest:~# zpool online -e bob c2t0d0
root@bhyvetest:~# zpool list bob
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
bob   21.5G  1.10M  21.5G        -         -     0%     0%  1.00x  ONLINE  -