Project

General

Profile

Actions
Copy link

Feature #14469

closed

nvme could raise dynamic lun expansion sysevents

Added by Andy Fiddaman over 1 year ago. Updated over 1 year ago.

Status:
Closed
Priority:
Low
Assignee:
Andy Fiddaman
Category:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Bite-size
Tags:
Gerrit CR:
1999
External Bug:

Description

I've been doing some testing of the new feature in bhyve as of #14372 where expanding the underlying storage for a virtual NVMe device causes an 'namespace changed' asynchronous event to be raised.

When this event occurs and the size of the namespace has increased, the driver could send a DLE (dynamic LUN expansion) sysevent so that other parts of the system (e.g. ZFS) can take appropriate action.

Related issues

Related to illumos gate - Feature #14372: bhyve upstream sync 2022 JanuaryClosedAndy Fiddaman

Actions
Related to illumos gate - Feature #14374: Improve bhyve mevent EVF_VNODE monitoringClosedAndy Fiddaman

Actions
Related to illumos gate - Feature #14471: vioblk could raise dynamic lun expansion syseventsClosedAndy Fiddaman

Actions
Related to illumos gate - Bug #14022: zpool online -e breaks access to poolClosedAndy Fiddaman

Actions
Related to illumos gate - Bug #14520: sysevent/zfs_mod should use C99ClosedAndy Fiddaman

Actions
Actions
Copy link
 #1

Updated by Andy Fiddaman over 1 year ago

  • Related to Feature #14372: bhyve upstream sync 2022 January added
  • Related to Feature #14374: Improve bhyve mevent EVF_VNODE monitoring added
Actions
Copy link
 #2

Updated by Andy Fiddaman over 1 year ago

Testing notes (using a slightly patched sysevent ZFS module which logs a couple of additional things):

root@bhyvetest:~# nvmeadm list
nvme1: model: bhyve-NVMe, serial: NVME-5-0, FW rev: 1.0, NVMe v1.3
  nvme1/589cfc20d5c30001 (c2t589CFC20D5C30001d0): Size = 10240 MB, Capacity = 10240 MB, Used = 10240 MB

root@bhyvetest:~# diskinfo
TYPE    DISK                    VID      PID              SIZE          RMV SSD
NVME    c2t589CFC20D5C30001d0   NVMe     bhyve-NVMe         10.00 GiB   no  yes

root@bhyvetest:~# zpool create tank c2t589CFC20D5C30001d0
root@bhyvetest:~# zpool get expandsz tank
NAME  PROPERTY    VALUE     SOURCE
tank  expandsize  -         -

root@bhyvetest:~# zdb | egrep 'tank:|phys_path|whole'
tank:
            phys_path: '/pci@0,0/pcifb5d,a0a@5/blkdev@w589CFC20D5C30001,0:a'
            whole_disk: 1

root@bhyvetest:~# svcadm disable -t sysevent
root@bhyvetest:~# /usr/lib/sysevent/syseventd -d9
syseventd[508]: syseventd started, debug level = 9
syseventd[508]: Pausing
syseventd[508]: client_deliver_event: id 0x95 size 208
received sysevent id = 0X2806ff3271:95
        class = EC_dev_status
        subclass = ESC_dev_dle
        vendor = SUNW
        publisher = nvme:0
nvlist version: 0
        phys_path = /devices/pci@0,0/pcifb5d,a0a@5/blkdev@w589CFC20D5C30001,0:a
...
syseventd[508]: zfsdle_vdev_online: searching for /pci@0,0/pcifb5d,a0a@5/blkdev@w589CFC20D5C30001,0:a in pool tank
syseventd[508]: zfsdle_vdev_online: found /dev/dsk/c2t589CFC20D5C30001d0s0 in pool tank (wd: 1)
syseventd[508]: Wholedisk, re-opening pool

root@bhyvetest:~# zpool get expandsz tank
NAME  PROPERTY    VALUE     SOURCE
tank  expandsize  1G        -

root@bhyvetest:~# diskinfo
TYPE    DISK                    VID      PID              SIZE          RMV SSD
NVME    c2t589CFC20D5C30001d0   NVMe     bhyve-NVMe         11.00 GiB   no  yes

... `zfs set volsize=30G` from outside the VM ...
root@bhyvetest:~# zpool get expandsz tank
NAME  PROPERTY    VALUE     SOURCE
tank  expandsize  20G       -

root@bhyvetest:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank   9.50G   960K  9.50G        -       20G     0%     0%  1.00x  ONLINE  -

root@bhyvetest:~# zpool online -e tank c2t589CFC20D5C30001d0

root@bhyvetest:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank   29.5G   996K  29.5G        -         -     0%     0%  1.00x  ONLINE  -
Actions
Copy link
 #3

Updated by Electric Monk over 1 year ago

  • Gerrit CR set to 1999
Actions
Copy link
 #4

Updated by Andy Fiddaman over 1 year ago

  • Related to Feature #14471: vioblk could raise dynamic lun expansion sysevents added
Actions
Copy link
 #5

Updated by Andy Fiddaman over 1 year ago

I did another test using the sysevent binary from Joyent's smartos-extra, with two pools based on NVMe disks inside a bhyve VM, while increasing the size of the volumes from the host OS.

root@bhyvetest:~# jobs
[1]+  Running                 ./sysevent -r &
root@bhyvetest:~#
root@bhyvetest:~#
root@bhyvetest:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  9.50G   953M  8.57G        -         -     0%     9%  1.00x  ONLINE  -
tank   9.50G   153K  9.50G        -         -     0%     0%  1.00x  ONLINE  -

nvlist version: 0
        date = 2022-02-15T11:45:39.210Z
        type = event
        vendor = SUNW
        publisher = blkdev
        class = EC_dev_status
        subclass = ESC_dev_dle
        pid = 0
        data = (embedded nvlist)
        nvlist version: 0
                phys_path = /devices/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC2045C20001,0:a
        (end data)

nvlist version: 0
        date = 2022-02-15T11:45:39.210Z
        type = event
        vendor = SUNW
        publisher = blkdev
        class = EC_dev_status
        subclass = ESC_dev_dle
        pid = 0
        data = (embedded nvlist)
        nvlist version: 0
                phys_path = /devices/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC2045C20001,0:b
        (end data)

root@bhyvetest:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  9.50G   951M  8.57G        -        5G     0%     9%  1.00x  ONLINE  -
tank   9.50G   153K  9.50G        -         -     0%     0%  1.00x  ONLINE  -

nvlist version: 0
        date = 2022-02-15T11:45:54.199Z
        type = event
        vendor = SUNW
        publisher = blkdev
        class = EC_dev_status
        subclass = ESC_dev_dle
        pid = 0
        data = (embedded nvlist)
        nvlist version: 0
                phys_path = /devices/pci@0,0/pcifb5d,a0a@5/blkdev@w589CFC20D5C30001,0:a
        (end data)

nvlist version: 0
        date = 2022-02-15T11:45:54.199Z
        type = event
        vendor = SUNW
        publisher = blkdev
        class = EC_dev_status
        subclass = ESC_dev_dle
        pid = 0
        data = (embedded nvlist)
        nvlist version: 0
                phys_path = /devices/pci@0,0/pcifb5d,a0a@5/blkdev@w589CFC20D5C30001,0:b
        (end data)

root@bhyvetest:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool  9.50G   951M  8.57G        -        5G     0%     9%  1.00x  ONLINE  -
tank   9.50G   153K  9.50G        -        5G     0%     0%  1.00x  ONLINE  -
Actions
Copy link
 #6

Updated by Andy Fiddaman over 1 year ago

  • Related to Bug #14022: zpool online -e breaks access to pool added
Actions
Copy link
 #7

Updated by Electric Monk over 1 year ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 6f0e4dc91b854250fff5c24de2d27aed3375ac69

commit  6f0e4dc91b854250fff5c24de2d27aed3375ac69
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2022-02-17T22:18:17.000Z

    14469 nvme could raise dynamic lun expansion sysevents
    Reviewed by: Robert Mustacchi <rm+illumos@fingolfin.org>
    Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions
Copy link
 #8

Updated by Andy Fiddaman over 1 year ago

  • Related to Bug #14520: sysevent/zfs_mod should use C99 added
Actions
Copy link

Also available in: Atom PDF